In previous posts I discussed some of the new features in PowerShell 7, such as Ternary Operators and Pipeline Chain Operators. Continuing with this series I wanted to now cover off the new Parallel feature in ForEach-Object.
ForEach-Object -Parallel <scriptblock> [-InputObject <psobject>] [-ThrottleLimit <int>] [-TimeoutSeconds <int>] [-AsJob]
[-WhatIf] [-Confirm] [<CommonParameters>]
There has been many ways to achieve Parallel operations in PowerShell prior to this feature so the concept isn’t anything new. In the case of ForEach-Object -Parallel, it leverages pre-existing APIs in PowerShell that have existed since way back in PowerShell v2, and gives us something a little easier to consume. It achieves this using something called Runspaces, where you’re able to create multiple threads running on the same process.
Why is this important to know. Well, a runspace comes with some overhead to use. It consumes memory, has to pass in variables and load modules. This all takes time to do and needs to be factored into what you’re trying to achieve. Used in the wrong situation it can very easily have a negative impact to use.
Let’s cover a very basic example below and see the effects when we try to parallelize it. The first is a simple ForEach-Object count execution running single threaded. It executes in around 3 seconds on my system.
$count = 1..5000
$count | ForEach-Object -Process { Write-Output "$_" }
In the second example I’ve replaced -Process with -Parallel. It executes in around 55 seconds on my machine and uses a staggering 4 GB of memory to complete.
$count = 1..5000
$count | ForEach-Object -Parallel {Write-Output "$_"}
Now this is by no means a scientific test. It’s more a, ‘your mileage may vary‘, situation. The point to highlight here is that just simply turning a ForEach-Object pipeline into parallel will not necessarily improve speed and performance.
So now that we’ve highlighted that ForEach-Object -Parallel is not a panacea to making everything run faster. We can start to cover off what you can do with it and a few situations where it may help speed up operations.
What if we have a script block in a ForEach-Object pipeline that takes some time to execute? For example Test-NetConnection, where we want to test a range of IPs, and each test will take anywhere up to about 20 seconds to complete.
Running the below single threaded will take around 5 minutes to complete on my machine.
$ip = 1..20
$ip | ForEach-Object -process {Test-NetConnection -ComputerName 10.0.0.$_ -WarningAction Ignore}
Running the same thing this time replacing -Process with -Parallel, it executes in 1 minute 15 seconds.
$ip = 1..20
$ip | ForEach-Object -parallel {Test-NetConnection -ComputerName 10.0.0.$_ -WarningAction Ignore}
-ThrottleLimit
This is where things get a little more interesting. When using the -Parallel parameter we can also a -ThrottleLimit parameter. This is basically the amount of runspaces that will attempt to run at the same time for the ForEach-Object script block. By default -Parallel uses a ThrottleLimit of 5. So we can try a new -ThrottleLimit of 10 this time and see what happens.
$ip = 1..20
$ip | ForEach-Object -Parallel {Test-NetConnection -ComputerName 10.0.0.$_ -WarningAction Ignore} -ThrottleLimit 10
This time it executes in 40 seconds. Great, right? Let’s just keep increasing the ThrottleLimit. Well, yes and no. You can try increasing the ThrottleLimit and may continue to see some gains. But there will also be a point where the overhead of creating so many runspaces outweighs the benefits. It’s recommended that -ThrottleLimit generally not exceed the number of CPU cores in the system.
$using:
Now what if we want to pass in multiple variables and not just the current value on the pipeline. Well we do this with the $using: keyword. We are required to do this for any variable other than the current value on the pipeline.
In the below example we define two variable, $Level and $MaxEvents. We use Get-WinEvent to get all event logs and we pass that down the pipeline to Foreach-Object -Parallel.
$Level = 'information'
$MaxEvents = '10'
Get-WinEvent -ListLog * -ErrorAction SilentlyContinue |
Where-Object { $_.recordcount -AND $_.lastwritetime -gt [datetime]::today } |
ForEach-Object -Parallel { Get-WinEvent -LogName $_.logname -MaxEvents $using:MaxEvents |
Where-Object { $_.leveldisplayname -eq $using:Level } }
When we reference our variables we specify $using:MaxEvents and $using:Level to call those values inside the runspaces. Anytime we want to reference the current value of the pipeline we can continue to use $_ as we normally would have in the past.
In summary the Parallel feature of ForEach-Object should be cautiously used. Script blocks that execute fast won’t really see any benefit. On the other hand script blocks that wait for something to complete could be good candidates to experiment with and parallelize.