Pull DP Upgrade Consideration in Large Environment

Many challenges come with managing and maintaining  System Center Configuration Manager in large environment. In previous posts, we covered how to limit Pull DP bandwidth and How to manage Pull DP with collections to ease management in large environment. Probably the biggest challenge with large environment,  is to keep SCCM up to date.

In this post, we will provide in depth details, considerations and useful tips on Pull DP upgrade for large SCCM environment. The goal is to provide ways to prevent upgrades of SCCM to become a nightmare by taking more time than planned and/or heavy hit on the network. We will cover theses subjects :

• Automatic package update and distribution
• How to work around automatic package update and distribution
• Pre-requisite checker
• Upgrade time lapse
• Distribution point and PullDP upgrade ratio
• What happen while Distribution point/ Pull DP upgrade

Note that all details and considerations are regarding major upgrade like service pack upgrades and major upgrades.

[su_note note_color=”#cbcbcb” radius=”8″]Warning : this post assume that you already have excellent SCCM knowledge. In Microsoft term, this could be target audience : level 300 :)[/su_note]

Automatic Package Updates and Distribution

After an upgrade, some packages will be automatically updated and distributed to all of your Distribution Point and Pull DP to provide the latest bits to be used by clients.

Here’s the list of what will be updated :

• The default SCCM client package (Around 200MB)

  

• Configuration Manager Client Upgrade Package (Around 2mb)
  • No control on this package

• Default Windows PE boot images  (Around 200MB for each boot image)
  • The update will happen no matter if ADK is updated or not
    • For example, we can see a change between R2 and R2 SP1 to the default background used in the WinPE
  • If it doesn’t automatically update, it means that it tried and failed because of driver injection
    • Manual update of the boot image is required to fix this issue, by removing faulty drivers

How to Work Around the Automatic Packages Update and Distribution

With the nature of packages automatically updated and distributed, it is possible to work around it prior to the upgrade of the site server.

Remove Boot Images from Distribution Points

Boot images, unlike SCCM client, can’t be pre-distributed on distribution point because of their unique nature.

To prevent a hit on the network, we removed the boot images from DP, Pull DP and DP groups.

This mean that OSD will NOT be available while the upgrade is being processed.

Once all DP/Pull DP are upgraded and system is heatlhy, redistribute boot image on all your DP/PullDP. This will give you total control on when to distribute instead of leaving the upgrade handle it.

Pre-Distribute SCCM Client Package Content

As describe above, the update of the SCCM client package will send around 200 Mb to each DP/ Pull DP.  To prevent a massive hit on the network, we created a standard package, before the upgrade, that has the content of the new SCCM client.

We got the source from another hierarchy (like a lab) that was already to the desired upgrade level.

Path to the client : SCCMInstallationDirectory\Client

The Content Library will only host a single instance of files, this will ease the distribution of the client after the upgrade as it will only validate and not distribute.

Prerequisite Checker

When the prerequisite checker runs, it will communicate with all DP and PullDP.

It takes approximately 5-10 seconds per PullDP… with 1000 PullDP, this could mean hours of waiting for the prerequisite checker to complete. Let it go, and follow the log for details.

You can follow the prerequisite checker in this log : C:\ConfigMgrPrereq.log

Upgrade Time Lapse

Upgrading a primary server will take time. Be patient.

As an example, upgrading a primary site server with 1000 Pull DP, 5000 clients, took more than 1 hour to complete the wizard.

Monitoring  Compmon.log after the upgrade wizard added another hour to the process. This log shows the various Site component status.

Compmon.log must show Waiting until the next polling cycle in 300 secondes from now to consider the upgrade completed.

This log can be find in : SCCMInstallationDirectory\Logs\compmon.log

Distribution Point and Pull Distribution Concurrent Upgrade Ratio

When upgrading a primary site, all child sites will be upgraded. In a large environment, DP and Pull DP are the biggest challenge of the upgrade.

We found this process not to be well documented and with many grey areas.

Change the DPUpgradeThreadLimit

By default, when upgrading a primary site, all Distribution and Pull DP will be upgraded.

As documented in this KB for DPUpgradeThreadLimit, the default limit is 5 concurrent Distribution Point or Pull DP Upgrade.

With 1000 Pull DP, this will take extremely long time. With a large SCCM environment, most probably the server and the network can handle more than 5 concurrent upgrade.

This setting can be changed for a higher rate of concurrent upgrade. We did it at 50 concurrent upgrade. Server and network did handle it quite well.

Here’s how to check if your DP settings :

Get the setting using a script querying the WMI of the primary server.

[pastacode lang=”javascript” message=”Check DP limit setting” highlight=”” provider=”manual”]

param(
[string] $siteServerName="."
)

$providerLocation = gcim -ComputerName $siteServerName -Namespace root\sms SMS_ProviderLocation -filter "ProviderForLocalSite='True'"
$providerMachine = $providerLocation.Machine
$sitecode = $providerLocation.SiteCode
$providerNamespace = "root\sms\site_" + $sitecode
$siteFilter = "SiteCode='" + $sitecode + "‘"

write-host $providerLocation



$distmgrConfig = gcim -ComputerName $providerMachine -Namespace $providerNamespace SMS_SCI_Component | ? {$_.ComponentName -eq "SMS_DISTRIBUTION_MANAGER"}

ForEach ($distMgrObject in $distmgrConfig)  {

    $properties = $distMgrObject | select -ExpandProperty Props
    $threadLimitProperty = $properties | ? {$_.PropertyName -eq "DPUpgradeThreadLimit"} 
    if($threadLimitProperty -eq $null)
    {
        write-host "Actual setting for DPUpgradeThreadLimit is using default (5)"
     
    }
    else
    {
        write-host "Actual setting for $($DistMgrObject.SiteCode) DPUpgradeThreadLimit is $($threadLimitProperty.Value)"
        }
    }

[/pastacode]

Here’s the script to change the setting in the WMI of the primary server.

[pastacode lang=”javascript” message=”Change DP/PullDP concurrent upgrade ratio” highlight=”” provider=”manual”]

param(
[string] $siteServerName=".",
#define the new number of DP/PullDP concurrent upgrade limit
[int] $newValue=50
)

$providerLocation = gcim -ComputerName $siteServerName -Namespace root\sms SMS_ProviderLocation -filter "ProviderForLocalSite='True'"
$providerMachine = $providerLocation.Machine
$sitecode = $providerLocation.SiteCode
$providerNamespace = "root\sms\site_" + $sitecode
$siteFilter = "SiteCode='" + $sitecode + "‘"

write-host $providerLocation



$distmgrConfig = gcim -ComputerName $providerMachine -Namespace $providerNamespace SMS_SCI_Component | ? {$_.ComponentName -eq "SMS_DISTRIBUTION_MANAGER"}

ForEach ($distMgrObject in $distmgrConfig)  {

    $properties = $distMgrObject | select -ExpandProperty Props
    $threadLimitProperty = $properties | ? {$_.PropertyName -eq "DPUpgradeThreadLimit"} 
    if($threadLimitProperty -eq $null)
    {
        write-host "Previous setting for DPUpgradeThreadLimit was using default, updating to $newValue"
        $newProperty = New-CimInstance -ComputerName $providerMachine -Namespace $providerNamespace -ClassName SMS_EmbeddedProperty
        $newProperty.PropertyName = "DPUpgradeThreadLimit"
        $newProperty.Value = $newValue#

        $newPropertyList = @()
        $properties | % { $newPropertyList += $_}
        $newPropertyList += $newProperty
    
        $distMgrObject.Props = $newPropertyList
        scim $distMgrObject
    }
    else
    {
        write-host "Previous setting for $($DistMgrObject.SiteCode) DPUpgradeThreadLimit was $($threadLimitProperty.Value), updating to $newValue"
        $newProperty.PropertyName = "DPUpgradeThreadLimit"
        $newProperty.Value = $newValue

        $newPropertyList = @()
        $properties | % { 
            if($_.PropertyName -eq "DPUpgradeThreadLimit")
            {
                $_.Value = $newValue
                $newPropertyList += $_
            }
            else
            {
                $newPropertyList += $_
           }  
        } 
    
        $distMgrObject.Props = $newPropertyList
        scim $distMgrObject 
       }
       }

[/pastacode]

Script credit : Matt Shadbolt

Here the result after modifying the WMI on our server.

Once modified in the WMI, restart the SMS_Executive so it takes effect.

What Actually Happen while Pull DP Upgrade

Pull DP need the same upgrade as the primary server.

The high-level steps are the following :

• Update package for Client, Client upgrade and Boot image, if already distributed
• Pull DP upgrade with PullDP.msi
• Automatically upgrade SCCM client (no matter what are the setting of Automatic Client upgrade)

In Details…

All those steps and observation occurs after the upgrade of the server. No action are required for the following to occur.

• First, it will trigger an Update Distribution Point on packages mentioned before:
  • Notice that the actual distribution will fail on all Pull Distribution point because they haven’t been upgraded yet
  • It took more than an hour to see the failure on all Pull DP

• All Pull DP will turn red in the distribution point configuration status

• While this happen, Pull DP Upgrade will be running 50 concurrent upgrade at a time
  • Monitor it through DistMgr.log

• Pull DP will start to report Upgrade successfully completed

• An SQL query will list the progress of the global PullDP Upgrade successfull

[su_note note_color=”#cbcbcb” radius=”8″]select * from v_DistributionPointMessages where MessageID=’2399′ order by LastStatusTime[/su_note]

• Once flagged as Upgraded successfull, it will automatically retry to send packages

• Slowly, Pull DP will be back to green state as distribution is successful. At one point, we had close to 2000 active distribution pending.

• Eventually, success distribution will go up
  • This took almost 48 hours to fully complete

• After the packages are sent, the client will automatically update to the new version

• Note that the client didn’t upgrade prior to this point because of the boundary configuration we had
  • The Pull DP was the only available Distribution point for itself

Hope this post will help you plan your next upgrade 🙂