Powershell: How to reduce runtime from 3 minutes when getting 22 ADGroups and 210 members

Question

I am generating a report of a subset of AD groups and their users, but to me the script takes a long time to finish. I have managed to reduce the runtime down to about 4 minutes from a previous version that needed about 13 minutes.

The script and its output is given below. I designed the script this way so that I just needed a single call to the server. I collect all the data I want in the Get-ADGroup call, and get attributes in the loops. The timings indicate that fetching all groups takes about 13 seconds, while the printing process takes almost 160 seconds. I expected the printing to be very quick, not more than a second.

The inner loop, the one iterating over the group members, needs about 7 seconds, regardless of how many members were in the group:

# Get all AD groups along with description and member data

$start_get = Get-Date
$adgroups = Get-ADGroup -server "" -Filter 'Name -like "Prefix*"' -Properties member,description
$stop_get = Get-Date
$dt_get = ($stop_get - $start_get).TotalSeconds
Write-Output "Time fetching groups: $dt_get s"

# Iterate over AD groups
$gstart = Get-Date
foreach ($group in $adgroups){
    # Write group data
    Write-Output "----------- Start of group -----------"
    Write-Output "Group Name        : $($group.name)"
    Write-Output "Group Description : $($group.description)"
    Write-Output "Group Members     :"

    # Iterate over group members
    $mstart = Get-Date
    foreach ($member in $group.member){
        # Extract the `CN` value in the member string by converting to a hash table
        # Credits: https://stackoverflow.com/a/67503277/6218849
        $name = ($member -split "," | ConvertFrom-StringData).CN
        Write-Output "    $name"
    }
    $mstop = Get-Date
    $dt_member = ($mstop - $mstart).TotalSeconds
    Write-Output "--------------------------------------"
    Write-Output "Time printing members: $dt_member s"
    Write-Output "------------ End of group ------------"
}
$gstop = Get-Date
$dt_print = ($gstop - $gstart).TotalSeconds
Write-Output "Time printing group: $dt_print s"
Write-Output "Total run time     : $($start_get + $gstop) s"

The output looks like this (minus the names and descriptions):

Time fetching groups: 14.8095232 s
----------- Start of group -----------
Group Name        :
Group Description :
Group Members     :
Population        : 10
--------------------------------------
Time printing members: 7.3385221 s
------------ End of group ------------
----------- Start of group -----------
Group Name        :
Group Description :
Group Members     :
Population        : 6
--------------------------------------
Time printing members: 7.2661961 s
------------ End of group ------------
----------- Start of group -----------
Group Name        :
Group Description :
Group Members     :
Population        : 2
--------------------------------------
Time printing members: 7.1176709 s
------------ End of group ------------
----------- Start of group -----------
Group Name        :
Group Description :
Group Members     :
Population        : 1
--------------------------------------
Time printing members: 7.0904087 s
------------ End of group ------------
----------- Start of group -----------
Group Name        :
Group Description :
Group Members     :
Population        : 3
--------------------------------------
Time printing members: 7.2423924 s
------------ End of group ------------
----------- Start of group -----------
Group Name        :
Group Description :
Group Members     :
Population        : 28
--------------------------------------
Time printing members: 7.3510665 s
------------ End of group ------------
----------- Start of group -----------
Group Name        :
Group Description :
Group Members     :
Population        : 8
--------------------------------------
Time printing members: 7.0845282 s
------------ End of group ------------
----------- Start of group -----------
Group Name        :
Group Description :
Group Members     :
Population        : 8
--------------------------------------
Time printing members: 7.0749375 s
------------ End of group ------------
----------- Start of group -----------
Group Name        :
Group Description :
Group Members     :
Population        : 54
--------------------------------------
Time printing members: 7.1177984 s
------------ End of group ------------
----------- Start of group -----------
Group Name        :
Group Description :
Group Members     :
Population        : 0
--------------------------------------
Time printing members: 7.0871232 s
------------ End of group ------------
----------- Start of group -----------
Group Name        :
Group Description :
Group Members     :
Population        : 10
--------------------------------------
Time printing members: 7.103549 s
------------ End of group ------------
----------- Start of group -----------
Group Name        :
Group Description :
Group Members     :
Population        : 7
--------------------------------------
Time printing members: 7.0192055 s
------------ End of group ------------
----------- Start of group -----------
Group Name        :
Group Description :
Group Members     :
Population        : 1
--------------------------------------
Time printing members: 7.0753799 s
------------ End of group ------------
----------- Start of group -----------
Group Name        :
Group Description :
Group Members     :
Population        : 14
--------------------------------------
Time printing members: 7.0894799 s
------------ End of group ------------
----------- Start of group -----------
Group Name        :
Group Description :
Group Members     :
Population        : 13
--------------------------------------
Time printing members: 7.0864959 s
------------ End of group ------------
----------- Start of group -----------
Group Name        :
Group Description :
Group Members     :
Population        : 0
--------------------------------------
Time printing members: 7.0609977 s
------------ End of group ------------
----------- Start of group -----------
Group Name        :
Group Description :
Group Members     :
Population        : 5
--------------------------------------
Time printing members: 7.0782862 s
------------ End of group ------------
----------- Start of group -----------
Group Name        :
Group Description :
Group Members     :
Population        : 6
--------------------------------------
Time printing members: 7.0750823 s
------------ End of group ------------
----------- Start of group -----------
Group Name        :
Group Description :
Group Members     :
Population        : 5
--------------------------------------
Time printing members: 7.2413143 s
------------ End of group ------------
----------- Start of group -----------
Group Name        :
Group Description :
Group Members     :
Population        : 4
--------------------------------------
Time printing members: 8.1748071 s
------------ End of group ------------
----------- Start of group -----------
Group Name        :
Group Description :
Group Members     :
Population        : 14
--------------------------------------
Time printing members: 7.4144964 s
------------ End of group ------------
----------- Start of group -----------
Group Name        :
Group Description :
Group Members     :
Population        : 11
--------------------------------------
Time printing members: 7.0616258 s
------------ End of group ------------
Time printing group: 158.3620035 s
Total run time     : 173.1715267 s

Data used in plotting:

[[10, 7.3385221],
 [6, 7.2661961],
 [2, 7.1176709],
 [1, 7.0904087],
 [3, 7.2423924],
 [28, 7.3510665],
 [8, 7.0845282],
 [8, 7.0749375],
 [54, 7.1177984],
 [0, 7.0871232],
 [10, 7.103549],
 [7, 7.0192055],
 [1, 7.0753799],
 [14, 7.0894799],
 [13, 7.0864959],
 [0, 7.0609977],
 [5, 7.0782862],
 [6, 7.0750823],
 [5, 7.2413143],
 [4, 8.1748071],
 [14, 7.4144964],
 [11, 7.0616258]]

Script for plotting:

import matplotlib.pyplot as plt
import pandas as pd

with open("timings.txt") as f:
    groups = f.read().split('----------- Start of group -----------\n')[1:]
    
    
data = []
for g in groups:
    p = int(g.splitlines()[3].split(":")[-1])
    t = float(g.splitlines()[5].split(":")[-1].split()[0])
    data.append([p, t])
    
df = pd.DataFrame(data, columns=["Population", "Timing"])
avg = df.groupby("Population").mean()

ax = avg.plot(figsize=(5, 3), marker=".", ms=2**4, mec="black", color="salmon", lw=4, mew=2, legend=None)
ax.set_xlabel("Number of members in group")
ax.set_ylabel("Runtime (seconds)")
ax.grid(ls=":", lw=0.75)
plt.tight_layout()
plt.savefig("Timings.png")

How do I make this faster than 3 minutes? Specifically, the part where I read member data from the group object. — Yoda, Dec 13 '22 at 15:43
there is no room for optimizing your script, you could try using pipelines and / or muiltithreading. AD outputs objects lazily, this is expected and it is how it works. aside, `-server ""` has nothing to do there — Santiago Squarzon, Dec 13 '22 at 16:01
This is where you need to start using parallel processing using [Parallel Runspaces in Powershell](https://mcpmag.com/articles/2017/08/24/parallel-runspaces-in-powershell.aspx). Also check [Multi-Threading in PS](https://learn.microsoft.com/en-us/powershell/scripting/learn/deep-dives/write-progress-across-multiple-threads?view=powershell-7.3) and [RUNSPACES](https://devblogs.microsoft.com/scripting/beginning-use-of-powershell-runspaces-part-1/) — Ranadip Dutta, Dec 13 '22 at 16:03
You can probably use LDAP search or filtering to do this faster. See: https://stackoverflow.com/questions/64176751/simple-script-to-list-members-of-a-group — Scepticalist, Dec 13 '22 at 16:46

Yoda · Answer 1 · 2022-12-14T10:13:46.613

Following the advice from @Santiago Squarzon and @Ranadip Dutta I looked into multithreading. The following solution is a straightforward adaptation of this answer. The ideal number of threads seem to be 4, which reduces the total runtime to about 28 seconds.

# Get all AD groups along with description and member data
# Using multithreading
# Credits: https://stackoverflow.com/a/43685953/6218849

# Get all AD groups with members and descriptions
$adgroups = Get-ADGroup -server "<myserver>" -Filter 'Name -like "Prefix*"' -Properties member,description

# Define block to be distributed over threads
$block = {
  Param ($adgroup)
  Write-Output "----------- Start of group -----------"
  Write-Output "Group Name   : $($adgroup.name)"
  Write-Output "Group Desc   : $($adgroup.description)"
  Write-Output "Group Members: "

  # Iterate over AD group members
  foreach ($member in $adgroup.member) {
    # Extract the `CN` value in the member string by converting to a hash table
    # Credits: https://stackoverflow.com/a/67503277/6218849
    $name = ($member -Split "," | ConvertFrom-StringData).CN
    Write-Output "    $name"
  }
  "------------ End of group ------------"
  Write-Output ""
}

# Clear job list
Get-Job | Remove-Job

# Spawn threads
$maxThreads = 4
foreach ($adgroup in $adgroups) {
  While ($(Get-Job -State Running).count -ge $maxThreads) {
    Start-Sleep -Milliseconds 3
  }
  Start-Job -ScriptBlock $block -ArgumentList $adgroup | Out-Null
}

# Wait for jobs to finish
While ($(Get-Job -State Running).count -gt 0) {
  Start-Sleep 1
}

# Get the data for each job
foreach ($job in Get-Job) {
  $data = Receive-Job -Id ($job.Id)
  Write-Output $data
}

# Clear job list
Get-Job | Remove-Job

Timings:

================
N   Runtime (s)
----------------
1   141.9
2    63.4
4    27.8
8    29.1
16   52.4
================

Powershell: How to reduce runtime from 3 minutes when getting 22 ADGroups and 210 members

1 Answers1