S2D Cluster Stuck in Degraded State

We ran into a problem this month with our S2D Hyper-V cluster.  Since the nodes replicate their disks, after a restart it needs to get the restarted node back in sync with the cluster.  This happens automatically, and we keep an eye on it to make sure the node pool is synced before we bring the restarted node back into production.

FYI – you can do this with:



However, this month the jobs were created, but never started.  I let it sit overnight, but when they still hadn’t run by the next morning.  I started to look into how to fix this.  Luckily, I found a blog with the right things to check and fix: http://kreelbits.blogspot.com/2017/06/storage-spaces-direct-storage-jobs-hung.html

He has a number of things to check, and what to do to fix them.  In our case the problem was that some of the drives stayed in maintenance mode.  From the Failover Cluster Manager they were not listed in maintenance, so without checking in PowerShell we probably never would have seen it.  To check the drives to see if any have an operational status of ‘In Maintenance Mode’ run


If you have some that aren’t ‘OK’, you can fix them by running

Get-PhysicalDisk | Where-Object { $_.OperationalStatus -eq "In Maintenance Mode" } | Disable-StorageMaintenanceMode

Once we did that the Storage jobs started and the drives got in sync.


Update Offline Virtual Machine

If you’ve created a golden image VM, you may have run across the hassle of keeping it updated.  You can fire it up monthly and run updates, or just wait and update as the first step in deploying a new machine.  OR…  You can deploy updates to the offline VM with a little prep work and powershell.

These notes are for deploying the monthly cumulative update on Windows Server 2016, but it should work for older versions and other updates.

The first step is to download the update file.  Then use the Expand program (built in) to extract the contents:


In the extract folder, find the .cab file for the update. Copy that to the VM host.  In this example, I’m using C:\Temp.

Now either open a PS Session or from the VM host open PowerShell.

Go to a folder on the local disk (this process failed when mounting into a CSV) and create a temporary folder to use as a mount point. It’s easiest if this is also where you’ve put the .cab file.

mkdir -Path C:\Temp\MountDir

Now mount the offline VHD(X) you want to update.

Mount-WindowsImage -ImagePath $VHDtoUpdate -Path C:\Temp\MountDir -Index 1

Now apply the image.

Add-WindowsPackage -Path C:\Temp\MountDir -PackagePath $updatepath

Wait while the process runs, then dismount the VHD(X).

Dismount-WindowsImage -Path C:\Temp\MountDir -Save

Now repeat for additional virtual machines, or clean up your temporary mount folder.

rd -Path C:\Temp\MountDir

Now you can start up your updated VM.

Full script:

$vmhost = <yourhost>
Enter-PSSession -ComputerName $vmhost

$cummupdatepath = "C:\Temp\Windows10.0-KB4022715-x64.cab"
$VHDtoUpdate = "C:\ClusterStorage\Volume1\Example\Virtual Hard Disks\Example.vhdx"

mkdir -Path C:\Temp\MountDir
Mount-WindowsImage -ImagePath $VHDtoUpdate -Path C:\Temp\MountDir -Index 1
Add-WindowsPackage -Path C:\Temp\MountDir -PackagePath $cummupdatepath
Dismount-WindowsImage -Path .\MountDir -Save
rd -Path C:\Temp\MountDir



Uninstall Application with Server Core

There is no way to directly uninstall a program with PowerShell, but you can get there in a PSRemote Session using WMI.  Note, this won’t work for Nano since it doesn’t have WMI.

First you need to open your connection and get the programs with WMI.

Enter-PSSession - ComputerName <YOURCOMPUTER>
$programs = Get-WmiObject -Class win32_product
$programs | Select Name

Now you can see the installed programs.  Find the one you want to remove, counting from zero to get its index in the array.  I want to remove program 5 in my example.  To test, I run a quick command to make sure I have the right entry.


If you get the right program back, proceed to uninstall.


You can rerun the initial commands to make sure it is no longer listed.  When you’re done, make sure to exit your PSRemote session.


Block SMB to your Workstations

When was the last time you had a business need to reach the C$ share on one of your workstations?  When was the last time you wanted a user workstation to reach the C$ share on another workstation?

If your answer was never or rarely, then you should block them.  It helps slow lateral movement of an attacker and if you were unpatched against MS17-010, but had this in place it would have prevented the spread of WannaCry.

This has been available since the introduction of the integrated firewall in Windows XP SP2.  It can be configured by Group Policy, so it’s easy to have it automatically applied to new machines as they are put into production.

Modify DPM Schedule with PowerShell

We recently ran into an issue where we wanted to update the backup schedules for a number of our protection groups.  I did one through the GUI and then decided to write up something in PowerShell.  This script is built to copy the settings from one protection group to another, so you do need to use the GUI once to set it up, or you could use these commands to do that initial change in PowerShell as well.

The odd thing is that you can’t actually copy a schedule from one to the other, so you put the schedule into a variable and pull out the pieces to copy it over.

$PG = Get-DPMProtectionGroup -DPMServerName <YOURDPMSERVER>
for ($i=0;$i -le ($pg.Count-1);$i++) {
Write-Output ('[' + $i.ToString() +'] ' + $pg[$i].Name)
$pgchoice1 = Read-Host -Prompt 'Choose Protection Group to copy schedule FROM'

$sched = Get-DPMPolicySchedule -ProtectionGroup $PG[$pgchoice1] -ShortTerm

for ($i=0;$i -le ($pg.Count-1);$i++) {
Write-Output ('[' + $i.ToString() +'] ' + $pg[$i].Name)
$pgchoice2 = Read-Host -Prompt 'Choose Protection Group to copy schedule TO'

$MPG = Get-DPMModifiableProtectionGroup -ProtectionGroup $PG[$pgchoice2]

Set-DPMPolicySchedule -ProtectionGroup $MPG -Schedule (Get-PolicySchedule $MPG -ShortTerm) -DaysofWeek $sched.WeekDays -TimesOfDay $sched.TimesOfDay

Set-DPMProtectionGroup -ProtectionGroup $MPG

Modify Replica Size with PowerShell

With DPM you may get errors that the DPM is out of disk space for the replica. (ID 58 Details: There is not enough space on the disk (0x80070070))  Assuming you’re not actually out of space, the replica size just needs to be increased.
Unfortunately, the menu option to Modify disk allocation isn’t selectable.
Luckily, we have PowerShell.  NOTE: These scripts assume you’re local to the DPM server, if not you’ll need to add on the -DPMServer switch to the initial commands.
First, we need to get the list of protection groups.  This script block pulls the groups into an array, then outputs their names with the array index.
$pg = Get-DPMProtectionGroup
for ($i=0;$i -le ($pg.Count-1);$i++) {
Write-Output ('[' + $i.ToString() +'] ' + $pg[$i].Name)
Now that you can get the name of the protection group, run the next script block with the index of that entry (in this example, I’m grabbing the second PG, or index 1).  The second bit outputs the datasources in the protection group with their index value in the array and the datasource name.
$ds = Get-DPMDatasource -ProtectionGroup $pg[1]
for ($i=0;$i -le ($pg.Count-1);$i++) {
Write-Output ('[' + $i.ToString() +'] ' + $ds[$i].Name)
Now you can grab the datasource you need to modify.  This script block grabs the individual datasource from the array of datasources, then outputs the name (to make sure you’ve got the right one) and then shows you the current replica size, in GB.
$d = $ds[0]
We can now modify the replica size.  Make sure you update the value for the ReplicaSize switch to match what you need.
Edit-DPMDiskAllocation -Datasource $d -ReplicaSize (24*1024*1024*1024)
Once that completes the Replica should have been resized.  You can use the UI to start a consistency check, but we’re in PowerShell, so let’s just do that here as well.
$cc = Start-DPMDatasourceConsistencyCheck -Datasource $d
Write-Output $cc.Status