An Attempt to initialize VM saved state failed – Could Not Create or Access Saved State File #HyperV #SCVMM

Hello Folks,

Well, after a few weeks away from work on vacation, I’ve finally managed to sit back and go through all my 30 Hyper-V hosts across different remote sites from a single pane of glass, of course you guessed it? yes, yes… Virtual Machine Manager Smile.

What I’ve come across from browsing through the console one Virtual Machine in a particular remote site is in a Stopped state.

And of course, what is the first thing you want to try to do is to power on that VM!

VMSS-Failed01

The unexpected behavior, the VM failed to start with the following error Surprised smile:

Error (12700)
VMM cannot complete the host operation on the hv01.dc.net server because of the error: ‘DC01′ could not initialize. (Virtual machine ID 7431245E-0FB3-45D3-B2E2-014124480F42)

‘DC01′ could not create or access saved state file D:\VirtualMachines\DC01\Virtual Machines\7431245E-0FB3-45D3-B2E2-014124480F42\7431245E-0FB3-45D3-B2E2-014124480F42.vsv. (Virtual machine ID 7431245E-0FB3-45D3-B2E2-014124480F42)
Unknown error (0×8001)

Recommended Action
Resolve the host issue and then try the operation again.

As we can see in the error above that the Virtual Machine could not access the saved state file .vsv.

The next logical thing is to remote into the Hyper-V host and try to dive deeper.

VMSS-Failed02

Again the VM failed to start.

VMSS-Failed05

As you can see in the above self-explanatory error that you do not have permission to perform this operation, therefore the issue is security permission.

Two errors are logged in the Event Viewer as well:

Event ID 3040:

VMSS-Failed03

Event ID 3080:

VMSS-Failed04

Back to basics, what files are used to create a virtual machine in Hyper-V:

  • .XML files
    • These files contain the virtual machine configuration details. There is one of these for each virtual machine and each checkpoint of a virtual machine. They are always named with the GUID used to internally identify the virtual machine
  • .BIN files
    • This file contains the memory of a virtual machine or checkpoint that is in a saved state.
  • .VSV files
    • This file contains the saved state from the devices associated with the virtual machine.
  • .VHD/VHDX files
    • These are the virtual hard disk files for the virtual machine.
  • .AVHD files
    • These are the differencing disk files used for virtual machine checkpoints aka (snapshots).
  • .VFD files
    • These are for virtual floppy disks files and rarely you use them Smile.

What are the symptoms that we observed so far:

  • Powering on the VM in Virtual Machine Manager, Hyper-V Manager or in PowerShell would result in an error  – could not create or access saved state file.
  • Error 3040 and 3080 was logged into the Hyper-V Worker event log.

I will browse into the .vsv and .bin files of the affected virtual machine and check it’s security permission.

.BIN:

VMSS-Failed06

.VSV:

VMSS-Failed07

If we compare the .vsv and .bin files for a running VM on the same host, we notice something different:

VMSS-Failed08

The GUID name of the virtual machine has full control on those files, but not on the affected VM.

One important point to mention that the GUID is the security context that is used to access the various files that make up the VM.  The VM Worker Process (vmwp.exe) will leverage this to work with the files. You can open up task manager or fire up PowerShell and see the VM Worker Process details, the GUID is listed in the user name field:

VMSS-Failed09

VMSS-Failed10

Let’s grant permissions to the .bin and .vsv files, you can use ICacls.exe from the command prompt (cmd):

The syntax is the following:

D:\VirtualMachines\DC01\Virtual Machine\7431245E-0FB3-45D3-B2E2-014124480F42\ICacls.exe 7431245E-0FB3-45D3-B2E2-014124480F42.bin /grant “NT VIRTUAL MACHINE\7431245E-0FB3-45D3-B2E2-014124480F42″:(F)

D:\VirtualMachines\DC01\Virtual Machine\7431245E-0FB3-45D3-B2E2-014124480F42\ICacls.exe 7431245E-0FB3-45D3-B2E2-014124480F42.vsv /grant “NT VIRTUAL MACHINE\7431245E-0FB3-45D3-B2E2-014124480F42″:(F)

Sure enough you need to adjust and match your GUID including the VM storage location.

Once I set the NTFS security permissions, I tried to power on the VM again!

And… Disappointed smile

VMSS-Failed05

Ehhhhh! What security permission is still missing Thinking smile

Let’s browse to the root folder of the affected Virtual Machine and compare the NTFS security permission with another running VM.

To illustrate this, you can fire up PowerShell and compare side by side using my favorite compare-object cmdlet:

PS C:\>$acl = Get-Acl -Path “D:\VirtualMachines\DC01″
PS C:\>$acl.access > C:\DC01.txt
PS C:\>$acl = Get-Acl -Path “D:\VirtualMachines\DHCP01″
PS C:\>$acl.access > C:\DHCP01.txt
PS C:\>Compare-object -referenceobject $(get-content C:\DHCP01.txt) -differenceobject $(get-content C:\DC01.txt)

VMSS-Failed11

What you can see in the SideIndicator outputis the NT Virtual Machines\Virtual Machines IdentityReference is available on the left arrow which means on the reference running VM ‘DHCP01’ and not available on the affected VM ‘DC01’.

If you look into NT Virtual Machines\Virtual Machines NTFS advanced security permission using the UI, you can see below:

VMSS-Failed12

Let’s grant the missing security permissions to the root DC01 Virtual Machine folder from ‘DHCP01’ using PowerShell.

PS C:\>$acl = Get-Acl -Path “D:\VirtualMachines\DHCP01″
PS C:\>Set-Acl -Path “D:\VirtualMachines\DC01″ -AclObject $acl
PS C:\Start-VM DC01
PS C:\Get-VM DC01

And here you go Smile

VMSS-Failed13

Conclusion

The best course of action is to never touch the security permission for the Virtual Machine files once they are working, and don’t change inheritance or modify any of these automatically added items, because when you first build a virtual machine, Hyper-V will set the security permission appropriately.

Hope this helps.

Enjoy your day!

Cheers,
/Charbel

Posted in Hyper-V, Security

Install and Configure VMM Network Builder #SCVMM #SysCtr #Cloud #HyperV

Hello Folks,

Good News! The System Center VMM Team just announced a new UI add-in tool to help customer on creating virtual network in Virtual Machine Manager.

If you are using Virtual Machine Manager in your environment and of course you should if you have more than 3 Hyper-V hosts to manage, then you examine the network fabric was the hardest part to configure Smile.

But…

The interns Program Managers on the System Center VMM team Mr. Anjay Ajodha and Mr. Matt McGlynn spent their summer analyzing the customer pain points regarding networking in VMM and have developed a small tool that should help ease the frustration with setting up VLAN-isolated networks in VMM. To help you get started quickly with networking in VMM and to simplify the process of creating new networks, they have created a new UI add-in in VMM home taskbar:

VMM-NB01

The entire basic networking setup can be created with this tool that can either be applied to hosts directly or used as a generic networking object base to be modified for customization to your configuration. VMM Network Builder compresses the steps required to build a logical switch. This tool will create networks that utilize VLAN isolation however it is not purposed for creating NVGRE networking configurations!

Let’s jump first into the installation of VMM Network Builder:

VMM-NB02

VMM-NB03

VMM-NB04

VMM-NB05

When the installation is done, you end up with NetworkBuilder.zip

VMM-NB08

Next, we need to import the Add-in, open VMM console and browse to Settings workplace and click on Import Console Add-in.

VMM-NB06

VMM-NB07

After the import is completed you can see an extra Icon in the title home bar “Build a Network“.

Let’s create and configure a basic VLAN-isolated network, click on “Build a Network“ and follow the wizard.

The Network Builder will connect to your VMM Server.

VMM-NB09

The first question is, do you to Create a Management Network?

VMM-NB10

The second option, do you have a separate Management NIC or all traffic going into the same physical NIC/Team?

In my environment, I am using Converged Network Fabric for (Management, Live Migration, Cluster, Virtual Machines, etc…), therefore I will select No.

VMM-NB11

Next you want to define the Management Network subnet.

VMM-NB12

Next step is to define a logical network name:

VMM-NB13

Now you want to define the Virtual Networks (VLAN ID and IP Pools).

VMM-NB14

Here a nice diagram showing the network configuration that will be created.

You can select “I would like to create a script to apply this logical switch to hosts”.

VMM-NB15

Here is the final summary before you create the VLAN-isolated network.

You can save the Script for future use as well.

VMM-NB16

As you noticed many different options are available in this tool, I recommend to try it and see how it works in your environment.

You can download the VMM Network Builder tool from here.

Enjoy your day!

Cheers,
/Charbel

Posted in Networking, System Center, Virtual Machine Manager

Free New eBook – Explored: 7 Key Areas of Hyper-V By Altaro Software #HyperV #Altaro

Hello Folks,

Here you go with a new free eBook gift from Altaro Software. The eBook is called Explored: 7 Key Areas of Hyper-V and it’s a detailed guide to help improve the core areas of a Hyper-V environment.

Altaro-eBook01

The eBook consists of 7 Chapters as the following:

Chapter 1: Seven Keys to Hyper-V Security
Manage Access to Virtual Machine Functions
Group Policy
File, Folder, and Share Security
The Network
The Guests
Antimalware
Patches and Hotfixes
Summary

Chapter 2: Hyper-V Manager – An Introduction
How to Acquire Hyper-V Manager
Enabling Hyper-V Manager
Interface Quick Tour
Differences between Hyper-V Manager and System Center Virtual Machine Manager
Hyper-V Cluster Integration
Failover Cluster Manager
Summary

Chapter 3: Set Up Native Net work Teams for Hyper-V
The GUI Way
The PowerShell Way
Related Cmdlets
Notes on the Windows Team
Link Aggregation and Bandwidth
Summary

Chapter 4: A Quick Guide to Hyper-V’s Virtual Switch
What You Get
The Fine Print
Summary

Chapter 5: Hyper-V Virtual CPUs
Physical Processors are Never Assigned to Specific Virtual Machines
Start by Understanding Operating System Processor Scheduling
Taking These Concepts to the Hypervisor
What about Processor Affinity?
How Does Thread Scheduling Work?
What Does the Number of vCPUs I Select Actually Mean?
But Can’t You Assign More Total vCPUs to all VMs than Physical Cores?
What’s The Proper Ratio of vCPU to pCPU/Cores?
What about Reserve and Weighting (Priority)?
But What About Hyper-Threading?
Summary

Chapter 6: Proper Use of Hyper-V Dynamic Disks
Terminology Clarification
What Dynamically Expanding Disks Are
FUD-Busting
How Dynamic VHD s Operate in the Real World
Making Fragmentation Go Away
Summary

Chapter 7: Connecting Hyper-V to Storage
Internal/Direct-Attached Disks
Prepare a Local Disk for Usage
Prepare a Storage Spaces Volume for Usage
Fibre Channel
iSCSI
Multi-Path I/O (MPIO)
SMB 3.0
Storage for a Hyper-V Cluster
Summary

Grab your free copy now @: ebook

Enjoy your day,

Cheers,
/Charbel

Posted in Altaro, Hyper-V

Hyper-V 2012 R2: Rollup Update, August 2014 #HyperV

Hello Folks,

As you already know that Microsoft pushed the second update this year for Windows 8.1 and Windows Server 2012 R2.

In this blog post I will list the new updates that applies only for Hyper-V Server 2012 R2.

Here you go 3 bug fixes for Hyper-V:

  • KB2972254: Hyper-V virtual machines cannot be connected to sometimes when TCP connections reconnect in Windows Server 2012 R2.

Symptoms: This issue occurs when a node of a failover cluster is removed from active cluster membership.

  • KB2978101: Windows 2012 R2-based Hyper-V host cluster freezes when virtual machines use shared virtual hard disks.

Symptoms: You have a Windows 2012 R2-based Hyper-V host cluster, or you create Hyper-V virtual machines on the cluster, and then you add shared virtual hard disk files (.vhdx) to the virtual machines.

  • KB2976884: “Access denied error” when Hyper-V Replica Broker goes online in a Windows Server 2012 or Windows Server 2012 R2 cluster.

Symptoms: You have a Windows Server 2012 R2 or Windows Server 2012 failover cluster that is in a domain, and the domain has a disjoint namespace.
You set the primary Domain Name Service (DNS) suffix of the Windows Server 2012 failover cluster to the disjoint domain name.
You create a Hyper-V Replica Broker in the failover cluster, and then you bring the Hyper-V Replica Broker online.

For detailed information visit: http://support.microsoft.com/kb/2975719

Patch, test, wait and then deploy in Production!!!

Happy Patching Smile

Cheers,
/Charbel

Posted in Hyper-V, Updates

What is VLAN Trunk Mode in Hyper-V? #HyperV

Hello Folks,

In today’s blog post we will dive into VLAN trunk mode in Hyper-V 2012 R2.

As you know, If you want to assign virtual machine network traffic to different VLANs, a network adapter that supports VLANs must be installed in the Hyper-V host and assigned to the virtual switch. You can set the VLAN ID for the virtual machine in the settings of the VM. The VLAN ID that is set in the virtual switch is the VLAN ID that will be assigned to the virtual network adapter (vmNIC) assigned to the host Operating System as showing in the following figure:

VLAN-ID01

VLAN ID settings for Virtual Machine. (Image: @CharbelNemnom)

What is VLAN trunk mode?
Hyper-V Virtual Switch also provides support for VLAN trunk mode. Trunk mode provides network services for a Virtual Machine with the ability to see traffic from multiple VLANs, in other words the virtual switch port receives traffic from all VLANs that you configure in an allowed VLAN list. You can also configure a switch port that is connected to a VM, but is not bound to the underlying NIC for trunk mode.

Configuring VLAN trunk mode for a (vmNIC) is not supported from Hyper-V Manager console; only a single VLAN can be configured as shown above.

So, how to do it?

With PowerShell, of course Smile

Set-VMNetworkAdapterVlan –VMName Demo –Trunk –AllowedVlanIdList 100-150 –NativeVlanId 20

In the example, “Demo” VM can send or receive traffic on any VLAN in the allowed list (100-150), and If there is no VLAN specified in the packet, the packet is treated as if it is from VLAN 20.

Note from the field
The NativeVlanId should not be within the scope listed in the AllowedVlanIdList (100-150), in other words if you set the Native VLAN ID to 110 for example, the VLAN 110 will be non functional, the VM itself will not be able to transmit tagged or untagged packets in VLAN ID 110.

Useful case
If you have a virtual machine that requires access to more networks than available virtual adapters, then you can enable VLAN trunk mode by using the Set-VMNetworkAdapterVlan cmdlet.

Enjoy your day!

Cheers,
/Charbel

Posted in Hyper-V, Network

Free Monitoring With Veeam Task Manager For Hyper-V (TMHV) #HyperV #Veeam

Hello Folks,

As you might noticed from the recent news, Veeam just announced and released Veeam Task Manager for Hyper-V (TMHV) free!

Here is quick overview:

As you already now that the default Task Manager built-in to Windows is not useful in a Hyper-V host because the CPU and memory usage shown does not show which VMs are using those resources. Veeam Task Manager for Hyper-V solves that problem with a unique on-demand real-time analysis of key performance counters for Hyper-V hosts and VMs running on these hosts as shown in below figure: 

TMHV-Veeam01

The system requirements for installation is Windows 7 and above including Hyper-V 2012 or 2012 R2.

The optimal installation is where? sure enough on your RSAT management machine Winking smile and then connect remotely to all your Hyper-V hosts and start monitoring like a PRO. Veeam Task Manager for Hyper-V use port TCP 445, make sure you open this port in the firewall on all hosts.

If you need more information about TMHV, you can find on Veeam’s website just announced by Mr. Mike Resseler here.

Give it a try and make your life easier, just register and download it from here.

Enjoy! Smile

Cheers,
/Charbel

Posted in Hyper-V, Monitoring

Building and Managing a Virtual Environment with Hyper-V Server 2012 R2 Video #HyperV @PacktPub @EricSiron

Hello Folks,

A New Hyper-V video training has been published by Packt Publishing last month, it’s named “Building and Managing a Virtual Environment with Hyper-V Server 2012 R2” authored by Mr. Eric Siron.

PacktPub-HyperV-Video01-Index

I am happy to announce that I was selected to review the training and post my feedback.

What you will learn from this video course:

  • Bring your Hyper-V nodes together into a cluster, remove failed nodes, and add new computer systems.
  • Explore the needs and the variety of storage options for a Hyper-V cluster.
  • Use the built-in graphical tools to manage your nodes and virtual machines.
  • Employ various methods to move virtual machines across nodes and clusters.
  • Use PowerShell and go beyond clustering to manage your nodes and virtual machines.
  • Gain a basic understanding of licensing Windows Server in a virtual environment.
  • Learn how to monitor the performance of your Hyper-V systems and meter virtual machines.
  • Get introduced to the cloud-enabling features of System Center Virtual Machine Manager.

And here is the table of content for your reference:

  1. Preparing the Nodes
    • Installing Hyper-V
    • Installing the Hyper-V PowerShell Module
    • Installing the Hyper-V Manager and Failover Cluster Manager
    • Network Adapter Teaming
    • The Hyper-V Virtual Switch
  2. Building the Cluster
    • Configuring Cluster Networks
    • Connecting to Storage
    • Cluster Validation
    • Cluster Creation
    • Adding Nodes and Storage to an Existing Cluster
  3. Virtual Machine Storage in a Cluster
    • Virtual Machine Files
    • Cluster Shared Volumes
    • Virtual Machines on SMB Shares
    • Storage and Share Permissions
    • Splitting Virtual Machine Storage
  4. Manipulating Virtual Machines in the GUI
    • Creating and Deleting Virtual Machines
    • Adjusting Memory Settings
    • Working with Hard Drives
    • Changing Network Settings
    • Manipulating Automatic Actions
  5. Manipulating Virtual Machines in PowerShell
    • Creating and Deleting Virtual Machines(PowerShell)
    • Adjusting Virtual Machine Memory(PowerShell)
    • Working with Hard Drives(PowerShell)
    • Changing Network Settings(PowerShell)
    • Manipulating Automatic Actions(PowerShell)
  6. Virtual Machine Mobility
    • Live Migrations
    • Quick Migrations
    • Failover
    • Storage Live Migrations
    • Shared Nothing Live Migrations
  7. Cluster Protection
    • Hyper-V Replica Setup
    • Hyper-V Replica Failover and Failback
    • Backup
    • Quality of Service
    • Cluster Aware Updating
  8. Cluster Maintenance
    • Licensing
    • Hyper-V Checkpoints
    • Hyper-V Best Practices Analyzer
    • Hyper-V Monitoring and Metering
    • Microsoft System Center Virtual Machine Manager

If you are novice or experienced Hyper-V, I would recommend you to get a copy of the training since the video course covers level 100 information up to level 300, and the course is 3 and a half hours, I am sure that you might find something interesting here.

Do you want to get a copy and save 15%?

The video course is available to download immediately from Packt Publishing website after registration: here.

Until next time,

Cheers,
/Charbel

Posted in Hyper-V, Video

How To Enable & Configure VMQ/dVMQ on Windows Server 2012 R2 with Below Ten Gig Network Adapters #HyperV #Vmq #vRSS

Hello Folks,

Back to basics: What is Virtual Machine Queue (VMQ), why do you need it and why you should enable it?

Virtual Machine Queue or dynamic VMQ is a mechanism for mapping physical queues in a physical NIC to the virtual NIC (vNIC) or virtual machine NIC (vmNIC) in Parent partition or Guest OS. This mapping makes the handling of network traffic more efficient. The increased efficiency results in less CPU time in the parent partition and reduced latency of network traffic.

VMQ spreads the traffic per vmNIC/vNIC, and each VMQ can use at most one logical CPU in the host, in other words VMQ distributes traffic equally amongst multiple guests (VMs) on a single host with a vSwitch (1 core per vmNIC/vNIC).

Note: The vNIC means a host partition Virtual NIC of the Virtual Switch in the Management OS, and the vmNIC is the synthetic NIC inside a Virtual Machine.    

VMQ is auto enabled by default on Windows Server machines when a vSwitch is created with 10Gig network adapters and above, and it’s useful when hosting many VMs on the same physical host.

The below figure is showing the ingress traffic with VMQ enabled for virtual machines.

image

[VMQ incoming traffic flow for virtual machines – source Microsoft]

When using 1Gig network adapters VMQ is disabled by default, because Microsoft don’t see any performance benefit to VMQ on 1Gig NICs, and one CPU/Core can keep up with 1Gig network traffic without any problem.

As I mentioned above with VMQ disabled all network traffic for vmNIC has to be handled by a single core/CPU, however with VMQ enabled and configured the network traffic is distributed across multiple CPUs automatically.

Now what happened if you have a large number of Web Servers VMs on a host with 2 eight core processors or more and with large amount of memory but you are limited by the physical NICs with 1Gig only?

The answer is…

VMQ and vRSS better together Smile

As I demonstrated in a previous blog posts, Post I and Post II, Windows Server 2012 R2 introduced a new feature called Virtual Receive Side Scaling (vRSS). This feature works with VMQ to distribute the CPU workload of receive network traffic across multiple (vCPUs) inside the VM. This effectively eliminates the CPU core bottleneck that we experienced with a single vmNIC. To take the full advantage of this feature both the host and the guest need to be Windows Server 2012 R2. As a result VMQ needs to be enabled on the physical host and RSS enabled inside the virtual machine, but until this point in time Microsoft don’t actually enable vRSS for the host vNICs, it’s only for VMs so we are stuck with one processor on the host Management partition with Converged Network environment. The good point is the vNICs on the host side get locked to one processor, but they will still get VMQs assuming you have enough Queues and they get distributed across different processors.  

The requirements to enable VMQ are the following:

1. Windows Server 2012 R2 (dVMQ+vRSS).
2. The Physical network adapters must support VMQ.
3. Install the latest NIC driver/firmware (very important).
4. Enable VMQ for 1Gig NICs in the registry, this step can be skipped if you have 10Gig adapters or more:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\VMSMP\Parameters\BelowTenGigVmqEnabled = 1
VmqBT-01

5. Reboot the host if you enable the registry key in step 4.
6. Determine the values for Base and Max CPU based on your hardware configuration.
7. Assign values for Base and Max CPU.
8. Enable RSS inside the Virtual Machines.
9. Turn on VMQ under Hyper-V settings for each VM which is already ON by default.

What is the Base CPU? It is the first CPU used for processing the incoming traffic for a particular vmNIC.

What is the Max CPU? It is the maximum number of CPU that we allow that NIC to process the traffic on.

Ok, so having this explained let’s configure VMQ step by step:

Our Lab Scenario:

We have 8 Physical 1Gig NICs and 2 X 8-core (32 logical processors).

First we need to determine if HyperThreading is enabled by running the following cmdlet:

PS C:\Get-WmiObject –Class win32_processor | ft –Property NumberOfCores, NumberOfLogicalProcessors –auto

VmqBT-02

As you can see we have the NumberOfLogicalProcessors as twice as the NumberOfCores, so we know that HT is enabled in the system.

Next, we need to look at our NIC Teaming and load distribution mode:

PS C:\Get-NetlbfoTeam | ft –Property TeamNics, TeamingMode, LoadBalancingAlgorithm –auto

VmqBT-03

After we determined that HyperThreading is enabled and the Teaming Mode is Switch Independent with Dynamic Mode, next we move on to Assign the Base and Max CPUs.

Attention: before moving into the assignment, one important point to consider, if the NIC team is in Switch-Independent teaming mode and the Load Distribution is set to Hyper-V Port mode or Dynamic mode, then the number of queues reported is the sum of all the queues available from the team members (SUM-Queues mode), otherwise the number of queues reported is the smallest number of queues supported by any member of the team (MIN-Queues mode).

What is (SUM-Queues mode) and What is (MIN-Queues mode)?

The SUM-Queues mode is the total number of VMQs of all the physical NICs that are participating in the team, however the MIN-Queues mode is the minimum number of VMQs of all the physical NICs that are participating in the team.

As an example, let’s consider we have two physical NICs with 4 VMQs each, if the teaming mode is Switch Independent with Hyper-V Port, the mode will be SUM-Queues equal to 8 VMQs, however if the teaming mode is Switch Dependent with Hyper-V Port the mode will be MIN-Queues equal to 4 VMQs.

[You can refer to the table below in order to determine the Teaming and Load distribution mode, source - Microsoft]:

Distribution mode→

Teaming mode↓

Address Hash modes Hyper-V Port Dynamic
Switch independent Min-Queues Sum-Queues Sum-Queues
Switch dependent Min-Queues Min-Queues Min-Queues

In our scenario, the NIC Team is in Switch Independent with Dynamic Mode so we are in SUM-Queues mode.

If the team is in Sum-Queues mode the team members’ processors should be, non-overlapping or with little overlap as possible. For example, in a 4-core host (8 logical processors) with a team of 2X10Gbps NICs, you could set the first NIC to use base processor of 2 and to use 4 cores; the second would be set to use base processor 6 and use 2 cores.

If the team is in Min-Queues mode the processor sets used by the team members must be identical, you should configure each NIC team member to use the same cores, in other words the assignment for each physical NIC will be the same.

Now let’s check first if VMQ is enabled:

PS C:\Get-NetAdapterVmq

VmqBT-04

As you can see VMQ is enabled (=True) but not yet configured.

And here we have two Converged Network Teams with 4 Physical NICs and 16 Queues each, so the total number of VMQs per team is 64.

I am using one Converged Team for vmNIC (VMs) and the second one is used for vNIC in the host.

We will set the Base and Max CPUs by running the following cmdlets for the teamedadapters under ConvergedNetTeam01:

PS C:\Set-NetAdapterVmq –Name NIC-b0-f0 –BaseProcessorNumber 2 –MaxProcessors 8
PS C:\Set-NetAdapterVmq –Name NIC-b0-f1 –BaseProcessorNumber 10 –MaxProcessors 8
PS C:\Set-NetAdapterVmq –Name NIC-b0-f2 –BaseProcessorNumber 18 –MaxProcessors 8
PS C:\Set-NetAdapterVmq –Name NIC-b0-f3 –BaseProcessorNumber 24 –MaxProcessors 8

As I mentioned above that in (Sum-Queues mode) you should configure the Base and Max CPU for each physical NIC with non-overlapping as possible, but in our lab environment we didn’t have as many cores as we had Queues so we had to have some overlap otherwise we are wasting our Queues.

Let’s run Get-NetAdapterVmq again and see the changes:

VmqBT-05

As you can see the Base and Max processors are set now, next we can run the Get-NetAdapterVmqQueue andthis will shows us how all queues are assigned across the VMQs in the vmNICs for all VMs on that particular host.

VmqBT-06

Now let’s see the result before and after VMQ + vRSS are enabled:

VMQ and vRSS disabled

In the Guest OS:

VmqBT-07

In the Host:

VmqBT-08

VMQ and vRSS enabled

In the Guest OS:

VmqBT-09

In the Host:

VmqBT-10

Last but not least best practices for configuring VMQ:

1. When using NIC Teaming, always use Switch Independent with Dynamic Mode when possible.
2. Make sure your base processor is never set to Zero to ensure best performance, because CPU0 handles special functions that cannot be handled by any other CPU in the system.
3. Keep in mind when assigning the Base/Max CPU and HyperThreading is enabled in the system, only the even number of processor is real processor (2,4,6,8, etc…), if HT is not enabled you can use even and odd number (1,2,3,4,5, etc…).
4. In SUM-Queues mode, try to configure the Base and Max CPU for each physical NIC with little overlap as possible, this is depends on the host hardware configuration with multiple cores.
5. Only assign Max Processor values of 1,2,4,8. It is ok to have a max processor number that will extend past the last core, or exceeds the number of VMQs on the physical NIC.
6. Don’t set the Base & Max processors on the Multiplexor NIC Teamed Adaptors, leave it as default.

In conclusion, I would prefer to enable VMQ on 1Gig NICs so I can keep my network traffic spread across as many CPU/cores as possible Smile.

For VMQ and RSS deep dive, here you go TechNet 3 part series VMQ Deep Dive.

Hope this helps.

Until then, enjoy your weekend!

Cheers,
/Charbel

Posted in Hyper-V, Network

Deploying HP StoreVirtual VSA On Hyper-V 2012 R2 Cluster – Part 3 #HyperV #HP #Storage #StoreVirtual #SysCtr

Hello Folks,

In Part I and Part II of this series we covered the deployment of HP StoreVirtual VSA including the Centralized Management Console (CMC) on Hyper-V, and the configuration of the StoreVirtual VSA Cluster and Failover Manager, Part III this post we will create our Hyper-V Cluster using Virtual Machine Manager.

So without further ado, let’s get started.

As I mentioned in Part I, HP StoreVirtual integrated very well with Microsoft Hyper-V such as HP Insight Control for Microsoft System Center, HP Management pack for SCOM, VSS provider and requestor functions, LeftHand DSM for MPIO, Application Aware Snapshot Manager, Recovery Manager for Windows, Windows Active Directory and SMI-S Support for System Center Virtual Machine Manager.

For streamlined workflows in Hyper-V environments, the storage presented by StoreVirtual VSA can be managed from within Microsoft System Center Virtual Machine Manager (SCVMM) 2012 SP1. Using VMM provision new storage for a virtual machine or rapid deployment of a virtual machine template with SAN Copy. HP StoreVirtual does not require a proxy agent, instead SCVMM communicates directly with the StoreVirtual Cluster.

Once the StoreVirtual cluster has been setup as described in Part II, the cluster virtual IP address of the VSA can be added as storage provider into Virtual Machine Manager. When adding StoreVirtual, make sure you use SSL encryption for the communication, default port (TCP 5989), and protocol SMI-S CIM-XML as showing in below screenshots:HPVSA-VMM02HPVSA-VMM04

HPVSA-VMM05

The storage resources need to be associated with a storage classification (administrator defined class of storage, such as Bronze, Silver, Gold).
HPVSA-VMM07

After the provider has been added successfully, all clusters in your StoreVirtual management group will be listed in the Classification and Pools list. New volumes can be provisioned from the available storage pools, used by new virtual machines and presented to Hyper-V hosts that are managed by Virtual Machine Manager via iSCSI.

Note: After you successfully registered the VSA in VMM via SMI-S, there is no information shown rather than the device name. No Vendor, No Model and even No storage information!!!HPVSA-VMM06

Unfortunately Microsoft System Center Virtual Machine Manager 2012 R2 is not supported yet by HP VSA SAN IQ11 Sad smile only VMM 2012 SP1 is qualified, and hopefully this will be fixed in the near future.

So let’s go back to the traditional way and present the LUN manually on each Hyper-V node via the iSCSI initiator.

After the LUN is mapped on each node, we will create Windows Server 2012 R2 Failover Cluster using the Hyper-V hosts in VMM. Any VMs that are deployed on the cluster will be made highly available, so after a host failure the VMs would automatically restart on another node in that cluster, providing resiliency to both planned and unplanned downtime.

Here are the steps to create the Cluster in VMM fairly simple:

1. Select the Fabric workspace in VMM.

2. Select the Home tab above the ribbon.

3. Click Create, and select Hyper-V Cluster.

HPVSA-VMM09

4. In the Create Cluster Wizard, on the General page, type your desired “Cluster Name” in the Name field.

5. Next to Use an existing Run As account, click Browse, select an admin account in order to create the cluster, and then click OK. Click Next.

HPVSA-VMM10

6. On the Nodes page, multi-select HV01 and HV02, click Add, and then click Next.

HPVSA-VMM12

7. On the IP Address page, check the box next to 172.16.20.0/24, select LAN IP Pool from the Static IP Pool dropdown. VMM will automatically allocate an IP address to the Cluster from that pool of IP Addresses. Click Next.HPVSA-VMM11

8. On the Storage page, select the disks you want to cluster, in my case the first disk will be selected as witness disk and the second one as Cluster Shared Volume (CSV).HPVSA-VMM13

9. On the Networks page, click Next. The Virtual Switches and Logical Switches for these hosts were already created.

10. On the Summary page, ensure the settings match below, and then click Finish.

11. Observe the running job in the Jobs window. Maximize the Jobs window. In the results pane at the bottom, select the Validate Cluster job, and click the Details tab. Observe the job through to completion.

12. This will take a few minutes to complete.

And here you go the Hyper-V Cluster is created!

HPVSA-VMM15

Hope this blog series demonstrated the full picture on how to deploy HP StoreVirtual VSA on Hyper-V and the integration with VMM.

If you missed Part I and Part II.

Thanks for reading.

Until then, enjoy your day!

Cheers,
/Charbel

Posted in Cluster, Hyper-V

Deploying HP StoreVirtual VSA On Hyper-V 2012 R2 Cluster – Part 2 #HyperV #HP #Storage #StoreVirtual #SysCtr

Hello Folks,

In Part I of this series we covered the deployment of HP StoreVirtual VSA and the Centralized Management Console (CMC) on Hyper-V, Part II this post is dedicated to the configuration of the StoreVirtual VSA Cluster and Failover Manager, in Part III we will create our Hyper-V Cluster using Virtual Machine Manager, I know it’s a long series blog posts, but I want to make sure that we cover all the aspects of the Hyper-V Software-Defined Storage deployment Smile.

So without further ado, we are in Step 3 in our deployment plan:

III- Configuration

Let’s start the Centralized Management Console (CMC). If you did not see any system under “Available Systems”, click “Find” on the menu and then choose “Find Systems…”. A dialog box will appear. Click “Add…” and enter the IP address of one of the earlier deployed VSA nodes (HP-VSA01). Repeat this until all deployed VSA nodes are added. Then click “Close”. Now you should have all available VSA nodes listed under “Available Systems”.

HP-CMC012

A management group is a collection of one or more storage VSA systems. It is the container within which your cluster storage systems and create volumes for storage. Creating a management group is the first step in creating HP StoreVirtual VSA Storage. Right click on any node and choose “Add to New Management Group…” from the context menu. We will add two nodes (HP-VSA01 and HP-VSA02) into this new management group.

HP-CMC013

As you can see warning message! To ensure that we have the highest level of availability we need to install HP StoreVirtual Failover Manager (FOM).

HP StoreVirtual FOM is a specialized version of the LeftHand OS designed to run as a virtual appliance in a virtual environment. The HP StoreVirtual FOM participates in the management group as a manager in the system performing quorum operations only, not data movement operations. It is recommended for 2-node or Multi-Site configurations to maintain quorum without requiring any additional physical hardware.

IMPORTANT: You must Install the Failover Manager on network hardware other than the storage systems in the SAN to ensure that it is available for failover and quorum operations if a storage system in the SAN becomes unavailable.

Now If you don’t use Failover Manager, and you select “I understand that not using FOM… a Virtual Manager is added to the management group by default, but is not started on a storage system until a failure in the system causes a loss of quorum. Unlike the Failover Manager, which is always running, the virtual manager must be started manually on a storage system after quorum is lost. It is designed to be used in two-systems or two-site system configurations which are at risk for a loss of quorum.

Note: A virtual manager requires manual intervention to recover quorum and can have undesirable effects when left running after quorum is recovered. Therefore, it’s highly recommended that you use the Failover Manager rather than the Virtual Manager.

OK, so having this explained and because I have only two VSA storage nodes, I will install HP StoreVirtual FOM on a third Hyper-V host and then repeat the cluster group wizard.

The installation of the Failover Manager for Hyper-V Server is straightforward:

Note: Before you start the installation, please make sure .NET Framework 3.5 (includes .NET 2.0 and 3.0) is installed on the Hyper-V host.

1. Insert the .iso image for HP StoreVirtual VSA and Failover Manager DVD.

2. Locate the applicable executable and double-click it to begin.HP-VSAFOM01

3. Click Agree to accept the terms of the License Agreement.HP-VSAFOM02
HP-VSAFOM03

4. Choose a location for the Failover Manager virtual machine and a location for the virtual hard
disks, and click Next.
HP-VSAFOM04

5. Enter the network information, including the host name and network addressing information, and click Next.HP-VSAFOM05

6. Enter a name for the Failover Manager, select whether you want it powered on after it is installed, and click Next.HP-VSAFOM06

7. Finish the installation, reviewing the configuration summary, and click Next. When the installer is finished, the Failover Manager is ready to be used in the HP StoreVirtual Storage.HP-VSAFOM07HP-VSAFOM08

After the installation, click FindFind Systems in the CMC and enter the Failover Manager IP address to discover the Failover Manager, and then add it to the management group, but before doing that let’s open Hyper-V Manager and check the settings of the StoreVirtual Failover Manager VM.HP-VSAFOM09

Now let’s create the Cluster Management Group again using the Centralized Management Console:HP-CMC014

Click “Next”. On the next page of the wizard we have to enter a username and password for the administrator user, that will be added to all VSA nodes.HP-CMC015

On the next page we need to provide NTP server. we can set the time manually, but a preferred NTP server is one that is more reliable, such as a server that is on a local network. An NTP server on a local network would have a reliable and fast connection to the storage system. They have to be reachable by the VSA nodes!HP-CMC016

On the next page of the wizard, you have to provide information about the DNS Server: DNS domain name, additional DNS suffixes and one or more DNS servers. For the DNS servers the same applies as for the NTP server. They have to be reachable by the VSA nodes!HP-CMC017

You can set up email notification for events for each management group. You must set up the email server to send events. You need the email (SMTP) server IP or host name, server port, and a valid email address to use as the sender address.HP-CMC018

Now very important question: Standard or Multi-Site Cluster? A Multi-Site cluster incorporates different features to ensure site fault tolerance. You must have the same number of VSA storage systems in each of the data sites spanned by your Multi-Site cluster. I chose to create a standard cluster since we don’t have Multi-Site deployment Winking smileHP-CMC019

After choosing the cluster type, we have to provide a cluster name and the number of nodes, that should be member of this new cluster.HP-CMC020

The next step is to configure the cluster Virtual IP address (Cluster VIP). This IP address has to be in the same subnet (VLAN) as the VSA nodes. This IP address is used to access the cluster. After the initial connection to the cluster VIP, the initiator will contact a VSA node for the data transfer.

A VIP is required for a fault-tolerant iSCSI cluster configuration, using VIP load balancing or the HP StoreVirtual DSM for Microsoft MPIO. When using a VIP, one storage system in the cluster hosts the VIP. All I/O goes through the VIP host. You can determine which storage system hosts the VIP by selecting the cluster, then clicking the iSCSI tab when the cluster is created.

Note: All iSCSI initiators (Hyper-V Hosts) must be configured to connect to the VIP address for the iSCSI failover to work properly.HP-CMC021

The wizard allows us to create a volume. This step can be skipped and create the volume later, however I created two full provisioned volumes, one volume 5GB for Windows Failover Cluster Quorum and another one 195GB for Cluster Shared Volume.

HP-CMC023HP-CMC025

After clicking “Finish” the management group and the cluster will be created. This steps could take some time.

At the end you will get a summary screen. You can create further volumes or you can repeat the whole wizard to create additional management groups or cluster.HP-CMC024

Congratulations! You have now a fully functional HP StoreVirtual VSA cluster Open-mouthed smile.

HP-CMC026

But wait we didn’t finish yet… Smile we need to add the Hyper-V Hosts.

To present the volumes to the hosts, you have to add hosts. A host consist of a name, an IP address, and an iSCSI IQN and, if needed, CHAP credentials. Multiple hosts can be grouped to server clusters. You need at least two hosts to build a server cluster. But first of all, we will add the two Hyper-V hosts:

HP-CMC028HP-CMC029

What is the Controlling Server IP Address to use?

When working with Hyper-V, it’s the Physical IP address required for each server hosting a VSS Provider in the Windows Failover Cluster. Since there is likely more than one LeftHand OS server in a Windows Failover Cluster, enter one physical IP address in each of the related LeftHand OS server dialogs.

With at least two hosts, you can create a server group. A server group simplifies the volume management, because you can assign and unassign volumes to a group of Hyper-V hosts with a single click. This ensures the consistency of volume presentations for a group of hosts.

HP-CMC030

Last but not least, we need to present the volumes, during the initial Cluster Management Group configuration we created two volumes 5GB and 195GB full-provisioned Network-RAID 10 volume. To assign the volumes to a server group, right-click the new server group in the CMC and click “Assign and Unassign Volumes…”. A window will popup and you can check or uncheck the volume and set the permission as Read-Only or Read-Write.

HP-CMC031

And here we go, we are nearly at the end. We only have to add the Cluster VIP to the iSCSI initiator on each Hyper-V host, initialize the disks and then create two volumes out of the presented volumes.

HP-CMC032

Part 3 will cover the configuration of the Hyper-V Cluster in VMM. If you have any question or feedback, please feel free to leave a comment below.

Thanks for reading.

Stay tuned… Until then, enjoy your day!

Cheers,
/Charbel

Posted in Cluster, Hyper-V
Follow

Get every new post on this blog delivered to your Inbox.

Join other followers: