No Available MAC Address for Dynamic Network Adapter in #WindowsServer 2016 #HyperV

| ,

Published on | Updated on January 26, 2021

4 Min. Read

Introduction

Since Hyper-V inception back in 2008, you can configure a virtual machine to use a dynamic or a static MAC address for any given network adapter.

By default, Hyper-V uses a dynamic MAC address – which means that Hyper-V will generate an initial MAC address for each network adapter, either for VM (vmNIC) or for the host (vNIC), and it will regenerate the MAC address if it believes it is necessary.

No Available MAC Address for Dynamic Network Adapter in #WindowsServer 2016 #HyperV 1

If you use static MAC addresses you need to manually specify the MAC address to use, but Hyper-V will never change it.

Each Hyper-V server has a MAC address range that it uses for generating new dynamic MAC addresses. You can also configure this range yourself if you want to.

The issue

I have two nodes Storage Spaces Direct (S2D) Hyper-Converged cluster, when creating a new VM on one of the nodes, I could not start that virtual machine. The error message is like this:

[An error occurred while attempting to start the selected virtual machine(s). Failed to Power on with Error ‘Attempt to access invalid address.’ (0x800701E7). No available MAC address for ‘Network Adapter’].

No Available MAC Address for Dynamic Network Adapter in #WindowsServer 2016 #HyperV 2

When checking the virtual network adapter for that virtual machine, I noticed that the MacAddress is set to zero 00:00:00:00:00:00

No Available MAC Address for Dynamic Network Adapter in #WindowsServer 2016 #HyperV 3

The only time you would expect to have a virtual machine with the MAC address set to zero is, if you have imported the virtual machine and specified that you are copying (and not moving) the virtual machine. This is not my case, I am creating a new VM.

No Available MAC Address for Dynamic Network Adapter in #WindowsServer 2016 #HyperV 4

The only time you would expect to receive No available MAC address for ‘Network Adapter’ is, if you are running more than 254 virtual machines. Because by default, each Hyper-V server can allocate a maximum of 254 MAC addresses. However, I have only 18 VMs running on that node!

I tried to set a static MAC Address for that VM and I was able to start it. What I found is, the affected node cannot start any VM with the Dynamic MAC address assigned. In other words, the Hyper-V server is not allocating any Dynamic MAC addresses for virtual machines.

On the second node, the VM is getting a Dynamic MAC and is starting without any issue. I was also able to Quick Migrate that VM and start it on the second node. Moreover, if I migrate a VM from the healthy node to the affected node, the VM is live migrating and starting successfully. Why? because the Dynamic MAC was moved along with VM from the healthy node.

So, what is wrong with that particular node?

Troubleshooting

The basic troubleshooting is as follows:

  1. Restart the Hyper-V Virtual Machine Management (VMMS) service. No Available MAC Address for Dynamic Network Adapter in #WindowsServer 2016 #HyperV 5
  2. Remove the VM network adapter, then add it again and try to start the virtual machine. No Available MAC Address for Dynamic Network Adapter in #WindowsServer 2016 #HyperV 6
  3. Restart the affected node.
  4. Remove MaximumMacAddress and MinimumMacAddress registry keys, and then restart the vmms service. Hyper-V will generate new Max and Min MAC addresses. No Available MAC Address for Dynamic Network Adapter in #WindowsServer 2016 #HyperV 7
  5. Expand the Maximum and Minimum MAC addresses range in Hyper-V. No Available MAC Address for Dynamic Network Adapter in #WindowsServer 2016 #HyperV 8
  6. Make sure that both nodes are running the same patch level. No Available MAC Address for Dynamic Network Adapter in #WindowsServer 2016 #HyperV 9

All the troubleshooting steps above did not lead anywhere. The virtual machine won’t start whatsoever.

Finding the cause

After digging in the Hyper-V logs, I’ve found a lot of errors related to Hyper-V-Worker, Hyper-V-SynthNic, and Hyper-V-VMMS.

No Available MAC Address for Dynamic Network Adapter in #WindowsServer 2016 #HyperV 10

Most of these errors were related to invalid access and permissions issues accessing the registry.

When I checked the registry permissions on the working node and on the affected node, I noticed a lot of access permissions were missing.

On the working node

No Available MAC Address for Dynamic Network Adapter in #WindowsServer 2016 #HyperV 11

On the affected node

No Available MAC Address for Dynamic Network Adapter in #WindowsServer 2016 #HyperV 12

As you can see, NT VIRTUAL MACHINE\Virtual Machines and NT SERVICE\TrustedInstaller permissions are missing from the list.

Fixing the issue

The first attempt is to try and set the right access permissions in the registry key by running the following PowerShell commands:

$ACL = Get-Acl -Path "HKLM:\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Virtualization"

$Inherit = [system.security.accesscontrol.InheritanceFlags]"ContainerInherit, ObjectInherit"
$Propagation = [system.security.accesscontrol.PropagationFlags]::None

$Rule = New-Object System.Security.AccessControl.RegistryAccessRule `
("NT Virtual Machine\Virtual Machines","FullControl",$Inherit,$Propagation,"Allow")

$Acl.AddAccessRule($Rule)
$Acl | Set-Acl

$Rule = New-Object System.Security.AccessControl.RegistryAccessRule `
("NT SERVICE\TrustedInstaller","FullControl",$Inherit,$Propagation,"Allow")

$Acl.AddAccessRule($Rule)
$Acl | Set-Acl

Get-Service VMMS | Restart-Service -Force

After I set the right access on the registry key, I tried to start the virtual machine, and this time I received a different message. The error is less descriptive now.

No Available MAC Address for Dynamic Network Adapter in #WindowsServer 2016 #HyperV 13

Unfortunately, the single remaining option in this scenario (which is more like a shotgun approach) is to do the following:

  1. Pause and drain the roles on the affected node, this will live migrate all virtual machines including cluster resources to the working node and won’t cause any downtime.
  2. Remove any vNIC that you are using on the host.
  3. Remove Hyper-V virtual switch including NIC Teaming (SET or Traditional LBFO Team).
  4. Uninstall Hyper-V role and then reboot the system.
  5. Delete the registry key: HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Virtualization on the affected node by running the following command. This command deletes the Virtualization registry key and all its subkeys and values.
    Get-Item -Path "HKLM:\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Virtualization" | Remove-Item -Recurse -Force -Verbose
  6. Reinstall Hyper-V role.
  7. Recreate Hyper-V virtual switch including NIC Teaming.
  8. Resume the node back to the cluster.
  9. Create virtual machines with Dynamic MAC address and you are good to go.

No Available MAC Address for Dynamic Network Adapter in #WindowsServer 2016 #HyperV 14

Summary

Before this issue occurred, the affected node crashed due to power loss. When the node came back online, it was not reachable over the network, all virtual NICs (vNICS) on the host was in disconnected mode, I was forced to remove and recreate the Switch Embedded Teaming (SET) virtual switch again. This caused to hit a timing issue on this one system which results in wrong permissions being applied to the registry key. Microsoft has seen this a handful of times and a fix is in place for the next release. As of this writing, the only option is to disable Hyper-V on the node, delete the registry key and then re-enable Hyper-V.

For more information about Hyper-V MAC Address allocation and network issues, I highly recommend reading an old article written by John Howard which is still valid today.

I want to thank Microsoft Program Manager Lars Iwer for supporting me in this case. Much obliged!

Hope this helps someone out there!

Cheers,
[email protected]

Previous

Technical Reviewer: Microsoft Official Courseware 20537 – Configuring and Operating a Hybrid Cloud with #Microsoft #AzureStack

What’s New in System Center Data Protection Manager – Preview 1711 #DPM #SCDPM #SysCtr

Next

Leave a comment below...

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Subscribe to Charbel Nemnom’s Blog

Get the latest posts delivered right to your inbox

The content of this website is copyrighted from being plagiarized! However, you can copy from the 'Code Blocks'.

Please send your feedback to the author using this form for any 'Code' you like.

Thank you for visiting!