Running Storage Spaces Direct With Nested Resiliency on HPE ProLiant MicroServers #S2D #WS2019

10 min read

Warning! This article is written for lab and demo purposes, it’s not supported in production.

Happy New Year folks!

Introduction

In Windows Server 2016, Microsoft introduced a new type of storage called Storage Spaces Direct (aka S2D). S2D enables building highly available storage systems with locally attached disks, and without the need to have any external SAS fabric such as shared JBODs or enclosures. This is the first true Software-Defined Storage (SDS) from Microsoft. Software-Defined Storage is a concept, which involves storing data without dedicated hardware.

In Windows Server 2019, Microsoft added a lot of improvements for Storage Spaces Direct. One of these improvements is a new type of resiliency known as nested resiliency that is designed only for two-server cluster and targeted for branch offices and small deployment.

With nested resiliency, we can stand multiple failures at the same time (we can lose one server and one drive, or two drives) compare to the classic two-way mirror (one server or one drive). Nested Resiliency also comes in two options:

  • Nested two-way mirror: This option is like a four-way mirror, with two copies in each server. The capacity efficiency ratio is ~25% versus 50% to the classic two-way mirror.
  • Nested mirror-accelerated parity: Combine nested two-way mirror with nested parity. The capacity efficiency ratio is around ~35% to 40%.

In this article, we will walk you through how to deploy Storage Spaces Direct in Hyper-Converged model with a nested two-way mirror on Windows Server 2019, and finally test the performance and compare it to the previous deployment with the classic two-way mirror on Windows Server 2016.

Hardware Configuration

2 X HPE ProLiant Micro Servers Gen 8, each system has the following set of specs:

  • 1 SATA SSD 2.5” 512 GB – model: SanDisk SD7SB2Q-512G-1006 (OSDisk)
  • 2 SATA SSD 2.5” 1 TB – model: Samsung SSD 840 EVO 1TB (S2D capacity storage)
  • 2 SATA SSD 2.5” 1 TB – model: Samsung SSD 860 EVO 1TB (S2D capacity storage)
  • 2 NIC 1 Gb Ethernet – model: Broadcom NetXtreme Gigabit Ethernet
  • 1 NIC 1 Gb Ethernet – model: Intel 82574L Gigabit Network Connection
  • 2 DDR3 8GB RAM 1600 MHz – 16 GB Total Memory
  • 1 Intel(R) Xeon(R) CPU E3-1265L V2 @ 2.50GHz 4/4 cores; 8 threads
  • 2 HPE Top of Rack Switches – model: PS1810-8G with Jumbo Frames and Spanning-tree enabled

The SSD drives used in this configuration are consumer grade and not intended to be used in production.

Software Configuration

We have the following set of software deployed for this demo:

  • Domain controller, DNS, and DHCP server
  • Host: Windows Server 2019 Datacenter Core Edition build number 10.0.17763 with December 2018 update
    • Single Storage Pool
    • 2 X 512 GB nested two-way mirror volumes
    • CSVFS_REFS file system
    • 10 virtual machines (5 VMs per node)
    • 2 virtual processors and 2 GB RAM per VM
    • VM: Windows Server 2016 Datacenter Core Edition with November 2018 update
    • Jumbo Frame enabled on all NICs

Network and Pre-Configuration

In the following steps, we will illustrate the network and pre-configuration steps:

  • Install Hyper-V and Failover Clustering roles. Set the necessary Windows firewall rules, enable remote desktop and set the power options to high performance. You can use the following set of PowerShell commands to automate this step:
# S2D hyper-converged cluster Pre-Configuration 
$Nodes = "S2D-HV01", "S2D-HV02"

Invoke-Command -ComputerName $Nodes -ScriptBlock {

#Install Hyper-V and Failover Cluster
Install-WindowsFeature Hyper-V, Failover-Clustering, FS-FileServer -IncludeAllSubFeature -IncludeManagementTools -Verbose

#Set Windows Firewall
Set-NetFirewallRule -Group "@firewallapi.dll,-36751" -Profile Any -Enabled true # Remote Shutdown firewall rule
Set-NetFirewallRule -DisplayName 'Windows Remote Management (HTTP-In)' -Profile Any -Enabled True -Direction Inbound -Action Allow
Set-NetFirewallRule -DisplayName 'Windows Management Instrumentation (WMI-In)' -Profile Any -Enabled True -Direction Inbound -Action Allow
Set-NetFirewallRule -DisplayName 'Remote Volume Management - Virtual Disk Service (RPC)' -Profile Any -Enabled True -Direction Inbound -Action Allow
Set-NetFirewallRule -DisplayName 'Remote Volume Management - Virtual Disk Service Loader (RPC)' -Profile Any -Enabled True -Direction Inbound -Action Allow
Set-NetFirewallRule -DisplayName 'File and Printer Sharing (Echo Request - ICMPv4-In)' -Enabled True -Direction Inbound -Action Allow -Profile Any
Set-NetFirewallRule -DisplayName 'File and Printer Sharing (Echo Request - ICMPv6-In)' -Enabled True -Direction Inbound -Action Allow -Profile Any
Set-NetFirewallRule -DisplayName 'File and Printer Sharing (SMB-In)' -Enabled True -Direction Inbound -Action Allow -Profile Any

# Enable Remote Desktop
(Get-WmiObject Win32_TerminalServiceSetting -Namespace root\cimv2\TerminalServices).SetAllowTsConnections(1,1) | Out-Null
(Get-WmiObject -Class "Win32_TSGeneralSetting" -Namespace root\cimv2\TerminalServices -Filter "TerminalName='RDP-tcp'").SetUserAuthenticationRequired(0) | Out-Null
Get-NetFirewallRule -DisplayName "Remote Desktop*" | Set-NetFirewallRule -enabled true

#Set the Windows Power plan to High Performance:
POWERCFG.EXE /S SCHEME_MIN

Restart-Computer -Force
}
  • In this step, we will set the cluster memory dump to “Active Memory Dump”. Starting with Windows Server 2016, Microsoft added a new option for creating memory dumps when a system failure occurs. The recommended setting for Failover Clustering is “Active memory dump“. This can be set in the “Startup and Recovery” dialog in the System control panel, Advanced system settings, or you can use the Set-ItemProperty cmdlet to specify the value of CrashDumpEnabled.
# Set Active Memory Dump on Server Core
Invoke-Command -ComputerName $Nodes -ScriptBlock {
# Configure Active memory dump
Set-ItemProperty –Path HKLM:\System\CurrentControlSet\Control\CrashControl –Name CrashDumpEnabled –value 1
New-ItemProperty –Path HKLM:\System\CurrentControlSet\Control\CrashControl -Name FilterPages -Value 1    
Get-ItemProperty –Path HKLM:\System\CurrentControlSet\Control\CrashControl 
} 
  • In this step, we will create Switch Embedded Teaming (SET) which is a new type of network teaming introduced in Windows Server 2016. We will create the management vNIC on the host as part of the converged SET virtual switch, we will set it’s bandwidth to weight and give it an IP address.
# Create S2D SET Team
Invoke-Command -ComputerName $Nodes -ScriptBlock {
    Param ($Nodes)
    $NICs = Get-NetAdapter
    New-VMSwitch -MinimumBandwidthMode Weight -NetAdapterName $NICs.Name[0],$NICs.Name[1],$NICs.Name[2] `
    -AllowManagementOS $true -EnableEmbeddedTeaming $true -Name "S2D_SET_vSwitch" -Notes "S2D_SET_vSwitch" -Verbose
    $vnic = Get-VMNetworkAdapter -ManagementOS
    Set-VMSwitch S2D_SET_vSwitch -DefaultFlowMinimumBandwidthWeight 30
    Set-VMSwitchTeam -Name "S2D_SET_vSwitch" -LoadBalancingAlgorithm HyperVPort
    
    Rename-NetAdapter -Name "vEthernet (S2D_SET_vSwitch)" -NewName "vEthernet (MGT_HostOS)"
    Set-VMNetworkAdapterVlan -VMNetworkAdapterName "S2D_SET_vSwitch" -Access -VlanId 0 -ManagementOS -Confirm:$false
    Set-VMNetworkAdapter -Name "S2D_SET_vSwitch" -MinimumBandwidthWeight 5 -ManagementOS

    If ($env:COMPUTERNAME -eq "$($Nodes[0])") {
    New-NetIPAddress -InterfaceAlias "vEthernet (MGT_HostOS)" -IPAddress 172.16.20.121 -PrefixLength 24 -DefaultGateway 172.16.20.1 -Type Unicast | Out-Null }
    Else {
    New-NetIPAddress -InterfaceAlias "vEthernet (MGT_HostOS)" -IPAddress 172.16.20.122 -PrefixLength 24 -DefaultGateway 172.16.20.1 -Type Unicast | Out-Null
       }
    
    Set-DnsClientServerAddress -InterfaceAlias "vEthernet (MGT_HostOS)" -ServerAddresses 172.16.20.9
} -ArgumentList $Nodes
  • In this step, we will create the host vNICs on the host partition as part of the converged SET virtual switch, we will set their IP addresses and bandwidth weight accordingly. We will also enable Virtual Receive Side Scaling (vRSS).
# Create Host vNICs
$switchName = "S2D_SET_vSwitch"
$LiveMigration = "172.20.42."
$Backup = "172.20.48."
$HVReplica = "172.20.46."
$Cluster =  "172.20.45."
$SMB_A = "172.20.50."
$SMB_B = "172.20.51."
$IP = 16

foreach ($Node in $Nodes) {
$MGT_LiveMigration = $LiveMigration + $IP
$MGT_Backup = $Backup + $IP
$MGT_HVReplica = $HVReplica + $IP
$MGT_Cluster = $Cluster + $IP
$MGT_SMB_A = $SMB_A + $IP
$MGT_SMB_B = $SMB_B + $IP
Invoke-Command -ComputerName $Node -ScriptBlock {
Param ($MGT_LiveMigration, $MGT_Backup, $MGT_HVReplica, $MGT_Cluster, $MGT_SMB_A, $MGT_SMB_B, $switchName)

Add-VMNetworkAdapter -SwitchName $switchName -ManagementOS -Name MGT_LiveMigration
New-NetIPAddress -InterfaceAlias "vEthernet (MGT_LiveMigration)" -IPAddress $MGT_LiveMigration -PrefixLength 24 -Type Unicast | Out-Null
Set-VMNetworkAdapterVlan -ManagementOS -VMNetworkAdapterName "MGT_LiveMigration" -Access -VlanId 42 -Confirm:$false
Set-VMNetworkAdapter -ManagementOS -Name "MGT_LiveMigration" -MinimumBandwidthWeight 20
Set-DNSClient -InterfaceAlias *Live* -RegisterThisConnectionsAddress $False

Add-VMNetworkAdapter -SwitchName $switchName -ManagementOS -Name MGT_Backup
New-NetIPAddress -InterfaceAlias "vEthernet (MGT_Backup)" -IPAddress $MGT_Backup -PrefixLength 24 -Type Unicast | Out-Null
Set-VMNetworkAdapterVlan -ManagementOS -VMNetworkAdapterName "MGT_Backup" -Access -VlanId 48 -Confirm:$false
Set-VMNetworkAdapter -ManagementOS -Name "MGT_Backup"  -MinimumBandwidthWeight 10
Set-DNSClient -InterfaceAlias *Backup -RegisterThisConnectionsAddress $False

Add-VMNetworkAdapter -SwitchName $switchName -ManagementOS -Name MGT_HVReplica
New-NetIPAddress -InterfaceAlias "vEthernet (MGT_HVReplica)" -IPAddress $MGT_HVReplica -PrefixLength 24 -Type Unicast | Out-Null
Set-VMNetworkAdapterVlan -ManagementOS -VMNetworkAdapterName "MGT_HVReplica" -Access -VlanId 46 -Confirm:$false
Set-VMNetworkAdapter -ManagementOS -Name "MGT_HVReplica" -MinimumBandwidthWeight 10
Set-DNSClient -InterfaceAlias *Replica -RegisterThisConnectionsAddress $False

Add-VMNetworkAdapter -SwitchName $switchName -ManagementOS -Name MGT_Cluster
New-NetIPAddress -InterfaceAlias "vEthernet (MGT_Cluster)" -IPAddress $MGT_Cluster -PrefixLength 24 -Type Unicast | Out-Null
Set-VMNetworkAdapterVlan -ManagementOS -VMNetworkAdapterName "MGT_Cluster" -Access -VlanId 45 -Confirm:$false
Set-VMNetworkAdapter -ManagementOS -Name "MGT_Cluster" -MinimumBandwidthWeight 5
Set-DNSClient -InterfaceAlias *Cluster -RegisterThisConnectionsAddress $False

Add-VMNetworkAdapter -SwitchName $switchName -ManagementOS -Name MGT_SMB_A
New-NetIPAddress -InterfaceAlias "vEthernet (MGT_SMB_A)" -IPAddress $MGT_SMB_A -PrefixLength 24 -Type Unicast | Out-Null
Set-VMNetworkAdapterVlan -ManagementOS -VMNetworkAdapterName "MGT_SMB_A" -Access -VlanId 50 -Confirm:$false
Set-VMNetworkAdapter -ManagementOS -Name "MGT_SMB_A" -MinimumBandwidthWeight 10
Set-DNSClient -InterfaceAlias *SMB_A -RegisterThisConnectionsAddress $False

Add-VMNetworkAdapter -SwitchName $switchName -ManagementOS -Name MGT_SMB_B
New-NetIPAddress -InterfaceAlias "vEthernet (MGT_SMB_B)" -IPAddress $MGT_SMB_B -PrefixLength 24 -Type Unicast | Out-Null
Set-VMNetworkAdapterVlan -ManagementOS -VMNetworkAdapterName "MGT_SMB_B" -Access -VlanId 51 -Confirm:$false
Set-VMNetworkAdapter -ManagementOS -Name "MGT_SMB_B" -MinimumBandwidthWeight 10
Set-DNSClient -InterfaceAlias *SMB_B -RegisterThisConnectionsAddress $False

Enable-NetAdapterRss -Name *
  } -ArgumentList $MGT_LiveMigration, $MGT_Backup, $MGT_HVReplica, $MGT_Cluster, $MGT_SMB_A, $MGT_SMB_B, $switchName
$IP++
}
  • Next, we will disable DNS registration for Storage, Cluster, Backup, Replica, and Live Migration vNICs by running the following PowerShell commands: Note: DNS registration should only be enabled on the Management Host vNIC.
# Disable DNS registration for Storage, Cluster, Backup and Live Migration network adapters by running the following commandlets:
Invoke-Command -ComputerName $Nodes -ScriptBlock {
Get-DnsClient | Where-Object {$_.InterfaceAlias -notmatch "OS" } | Set-DNSClient -RegisterThisConnectionsAddress $false
}
  • In the final step, we will enable Jumbo Frames on all vNICs by running the following PowerShell commands:
# Configure Jumbo Frame on each network adapter
Invoke-Command -ComputerName $Nodes -ScriptBlock {
Get-NetAdapterAdvancedProperty -Name * -RegistryKeyword "*jumbopacket" | Set-NetAdapterAdvancedProperty -RegistryValue 9014 
Get-NetAdapterAdvancedProperty -Name * -RegistryKeyword "*jumbopacket" | FT -AutoSize
} 

Create S2D Cluster

In the following steps, we will create the S2D cluster, but before doing so we will validate the cluster support:

  • Open Windows PowerShell and run the following command:
Test-Cluster -Node $Nodes -Include Inventory, Network, "System Configuration", "Storage Spaces Direct" -Verbose

  • Once the cluster validation is succeeded, we will move into creating the cluster by running the following command.
# New S2D Cluster
$Cluster = "NINJA-S2DCLU"
New-Cluster -Name $Cluster -Node $Nodes -NoStorage -StaticAddress 172.16.20.120/24 -Verbose
# Configure File Share Witness
Set-ClusterQuorum -Cluster $Cluster -FileShareWitness \\172.16.20.152\USBDisk1 -Credential $(Get-Credential)

Enable Storage Spaces Direct With Nested Resiliency

In the following steps, we will enable Storage Spaces Direct:

  • Open Windows PowerShell and enable Storage Spaces Direct by running the following command. In this example, we will disable the Cache since we are using a full flash system.
Enable-ClusterS2D -CimSession $Cluster -PoolFriendlyName "NINJA-S2D-HVPOOL" -Confirm:$false -CacheState Disabled -Verbose
Get-ClusterS2D -CimSession $Cluster
  • As mentioned earlier, we are using a full flash system without cache device in this deployment, the remaining four drives in each node will be used as a capacity tier. We can look at the drives inventory in Windows Admin Center.

  • When you enable Storage Spaces Direct, S2D will automatically create two storage tiers, known as performance tier and capacity tier. Enable-ClusterS2D cmdlet analyzes the devices and configures each tier with the mix of device types and resiliency (Mirror and Parity). In other words, the storage tier details and resiliency depend on the storage devices in the system and thus vary from system to system. However, the storage tier templates for Nested Resiliency are not created by default. First, we will create the new storage tier templates for mirror and parity by using the New-StorageTier cmdlet and then specify the Media Type either HDD or SSD. In this example, we are using only Solid-State Drives (SSD). At the time of this writing, PowerShell is the only way to create volumes with nested resiliency. To create the storage tier templates, you can use the following PowerShell commands:
# For mirror SSD
New-StorageTier -CimSession $Cluster -StoragePoolFriendlyName *S2D* -FriendlyName NestedMirror -ResiliencySettingName Mirror -MediaType SSD -NumberOfDataCopies 4

# For parity SSD
New-StorageTier -CimSession $Cluster -StoragePoolFriendlyName *S2D* -FriendlyName NestedParity -ResiliencySettingName Parity -MediaType SSD -NumberOfDataCopies 2 `
-PhysicalDiskRedundancy 1 -NumberOfGroups 1 -FaultDomainAwareness StorageScaleUnit -ColumnIsolation PhysicalDisk
  • To verify that the tiers for nested resiliency are created successfully, you can use the Get-StorageTier cmdlet.

  • To figure out the supported storage size for each tier (NestedMirror and NestedParity), you can use the following PowerShell commands.
# Get Supported storage size for NestedMirror tier
Get-StorageTierSupportedSize -FriendlyName NestedMirror -CimSession $Cluster | `
Select @{l="TierSizeMin(GB)";e={$_.TierSizeMin/1GB}}, @{l="CapacityTierSizeMax(TB)";e={$_.TierSizeMax/1TB}}, @{l="TierSizeDivisor(GB)";e={$_.TierSizeDivisor/1GB}}

# Get Supported storage size for NestedParity tier
Get-StorageTierSupportedSize -FriendlyName NestedParity -CimSession $Cluster | `
Select @{l="TierSizeMin(GB)";e={$_.TierSizeMin/1GB}}, @{l="CapacityTierSizeMax(TB)";e={$_.TierSizeMax/1TB}}, @{l="TierSizeDivisor(GB)";e={$_.TierSizeDivisor/1GB}}
  • As you can see in the next screenshot and based on this example, we have around 1.8 TB capacity for NestedMirror tier and 2.4 TB for NestedParity tier. If we compare the capacity efficiency with regular two-way mirror versus nested two-way mirror, we get 50% efficiency (~3.6 TB), while we get 25% (~1.8 TB) with a nested two-way mirror. As a side note, nested two-way mirror achieves capacity efficiency of 25% which is the lowest of any resiliency option in Storage Spaces Direct, however, nested parity achieves higher capacity efficiency, around 35%-40% and that depends on two factors: the number of capacity drives in each server, and the mix of mirror and parity you specify when creating the volume.

  • In this example, we will create two nested two-way mirror volumes (one per node) which give us higher performance than nested parity. To use a nested two-way mirror, you need to use the New-Volume cmdlet with reference NestedMirror tier template as shown in the following example and then specify the size of the volume.
# Created Nested two-way mirror 512GB
$Nodes = "NINJA-S2D-HV01", "NINJA-S2D-HV02"
Foreach ($Node in $Nodes) {
New-Volume -CimSession $Cluster -FriendlyName $Node -StoragePoolFriendlyName *S2D* -StorageTierFriendlyNames NestedMirror `
-StorageTierSizes 512GB -FileSystem CSVFS_ReFS -Verbose
}
  • Please note that volumes that use nested resiliency appear in Windows Admin Center with clear labeling, as shown in the screenshot below. Microsoft is moving away from the traditional Failover Cluster Manager (FCM) console, you cannot see nested resiliency in FCM. Once the volumes are created, you can manage and monitor them using Windows Admin Center just like any other volume in Storage Spaces Direct.

Workload Test Configuration

  • DISKSPD version 2.0.21 workload generator
  • VMFleet workload orchestrator

Total 160K IOPS – Read Latency @ 0.2ms and Write Latency @ 3ms

Each VM is configured with:

4K IO size
10GB working set
100% read and 0% write
No Storage QoS
No RDMA
No deduplication

Total 12K IOPS – Read Latency @ 32ms and Write Latency @ 73ms

Each VM is configured with:

4K IO size
10GB working set
70% read and 30% write
No Storage QoS
No RDMA
No deduplication

Summary

In this article, we showed you how to enable Storage Spaces Direct with nested two-way mirror resiliency on two nodes using HPE ProLiant MicroServers. This setup is used for lab and test environment only and it’s not supported for production. For more information about Nested Resiliency, please check the Microsoft documentation here.

There are two factors to consider when using nested resiliency in Windows Server 2019. First, you will see lower write IOPS than the classic two-way mirror, with 100% read, we got pretty similar performance to the previous deployment on the same hardware with Windows Server 2016, this because all reads are served locally from within the node, however, we saw very poor performance with 30% write since nested resiliency has to write an additional copy on each node. Second, you will have less usable capacity to use for production workloads, but with nested resiliency what you gain in return is higher uptime and availability. Storage is cheap, but downtime is expensive!!!

I highly recommend checking a great article by my fellow MVP and dear friend Darryl van der Peijl about the comparison between Two-way mirror and nested resiliency.

__
Thank you for reading my blog.

If you have any questions or feedback, please leave a comment.

-Charbel Nemnom-

Advertisements
About Charbel Nemnom 475 Articles
Charbel Nemnom is a Cloud Architect and Microsoft Most Valuable Professional (MVP), totally fan of the latest's IT platform solutions, accomplished hands-on technical professional with over 17 years of broad IT Infrastructure experience serving on and guiding technical teams to optimize performance of mission-critical enterprise systems. Excellent communicator adept at identifying business needs and bridging the gap between functional groups and technology to foster targeted and innovative IT project development. Well respected by peers through demonstrating passion for technology and performance improvement. Extensive practical knowledge of complex systems builds, network design and virtualization.

Be the first to comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.