An Attempt to initialize VM saved state failed – Could Not Create or Access Saved State File #HyperV #SCVMM

Hello Folks,

Well, after a few weeks away from work on vacation, I’ve finally managed to sit back and go through all my 30 Hyper-V hosts across different remote sites from a single pane of glass, of course you guessed it? yes, yes… Virtual Machine Manager Smile.

What I’ve come across from browsing through the console one Virtual Machine in a particular remote site is in a Stopped state.

And of course, what is the first thing you want to try to do is to power on that VM!

VMSS-Failed01

The unexpected behavior, the VM failed to start with the following error Surprised smile:

Error (12700)
VMM cannot complete the host operation on the hv01.dc.net server because of the error: ‘DC01’ could not initialize. (Virtual machine ID 7431245E-0FB3-45D3-B2E2-014124480F42)

‘DC01’ could not create or access saved state file D:\VirtualMachines\DC01\Virtual Machines\7431245E-0FB3-45D3-B2E2-014124480F42\7431245E-0FB3-45D3-B2E2-014124480F42.vsv. (Virtual machine ID 7431245E-0FB3-45D3-B2E2-014124480F42)
Unknown error (0x8001)

Recommended Action
Resolve the host issue and then try the operation again.

As we can see in the error above that the Virtual Machine could not access the saved state file .vsv.

The next logical thing is to remote into the Hyper-V host and try to dive deeper.

VMSS-Failed02

Again the VM failed to start.

VMSS-Failed05

As you can see in the above self-explanatory error that you do not have permission to perform this operation, therefore the issue is security permission.

Two errors are logged in the Event Viewer as well:

Event ID 3040:

VMSS-Failed03

Event ID 3080:

VMSS-Failed04

Back to basics, what files are used to create a virtual machine in Hyper-V:

  • .XML files
    • These files contain the virtual machine configuration details. There is one of these for each virtual machine and each checkpoint of a virtual machine. They are always named with the GUID used to internally identify the virtual machine
  • .BIN files
    • This file contains the memory of a virtual machine or checkpoint that is in a saved state.
  • .VSV files
    • This file contains the saved state from the devices associated with the virtual machine.
  • .VHD/VHDX files
    • These are the virtual hard disk files for the virtual machine.
  • .AVHD files
    • These are the differencing disk files used for virtual machine checkpoints aka (snapshots).
  • .VFD files
    • These are for virtual floppy disks files and rarely you use them Smile.

What are the symptoms that we observed so far:

  • Powering on the VM in Virtual Machine Manager, Hyper-V Manager or in PowerShell would result in an error  – could not create or access saved state file.
  • Error 3040 and 3080 was logged into the Hyper-V Worker event log.

I will browse into the .vsv and .bin files of the affected virtual machine and check it’s security permission.

.BIN:

VMSS-Failed06

.VSV:

VMSS-Failed07

If we compare the .vsv and .bin files for a running VM on the same host, we notice something different:

VMSS-Failed08

The GUID name of the virtual machine has full control on those files, but not on the affected VM.

One important point to mention that the GUID is the security context that is used to access the various files that make up the VM.  The VM Worker Process (vmwp.exe) will leverage this to work with the files. You can open up task manager or fire up PowerShell and see the VM Worker Process details, the GUID is listed in the user name field:

VMSS-Failed09

VMSS-Failed10

Let’s grant permissions to the .bin and .vsv files, you can use ICacls.exe from the command prompt (cmd):

The syntax is the following:

D:\VirtualMachines\DC01\Virtual Machine\7431245E-0FB3-45D3-B2E2-014124480F42\ICacls.exe 7431245E-0FB3-45D3-B2E2-014124480F42.bin /grant “NT VIRTUAL MACHINE\7431245E-0FB3-45D3-B2E2-014124480F42”:(F)

D:\VirtualMachines\DC01\Virtual Machine\7431245E-0FB3-45D3-B2E2-014124480F42\ICacls.exe 7431245E-0FB3-45D3-B2E2-014124480F42.vsv /grant “NT VIRTUAL MACHINE\7431245E-0FB3-45D3-B2E2-014124480F42”:(F)

Sure enough you need to adjust and match your GUID including the VM storage location.

Once I set the NTFS security permissions, I tried to power on the VM again!

And… Disappointed smile

VMSS-Failed05

Ehhhhh! What security permission is still missing Thinking smile

Let’s browse to the root folder of the affected Virtual Machine and compare the NTFS security permission with another running VM.

To illustrate this, you can fire up PowerShell and compare side by side using my favorite compare-object cmdlet:

PS C:\>$acl = Get-Acl -Path “D:\VirtualMachines\DC01”
PS C:\>$acl.access > C:\DC01.txt
PS C:\>$acl = Get-Acl -Path “D:\VirtualMachines\DHCP01”
PS C:\>$acl.access > C:\DHCP01.txt
PS C:\>Compare-object -referenceobject $(get-content C:\DHCP01.txt) -differenceobject $(get-content C:\DC01.txt)

VMSS-Failed11

What you can see in the SideIndicator outputs the NT Virtual Machines\Virtual Machines IdentityReference is available on the left arrow which means on the reference running VM ‘DHCP01’ and not available on the affected VM ‘DC01’.

If you look into NT Virtual Machines\Virtual Machines NTFS advanced security permission using the UI, you can see below:

VMSS-Failed12

Let’s grant the missing security permissions to the root DC01 Virtual Machine folder from ‘DHCP01’ using PowerShell.

PS C:\>$acl = Get-Acl -Path “D:\VirtualMachines\DHCP01”
PS C:\>Set-Acl -Path “D:\VirtualMachines\DC01” -AclObject $acl
PS C:\Start-VM DC01
PS C:\Get-VM DC01

And here you go Smile

VMSS-Failed13

Conclusion

The best course of action is to never touch the security permission for the Virtual Machine files once they are working, and don’t change inheritance or modify any of these automatically added items, because when you first build a virtual machine, Hyper-V will set the security permission appropriately.

Hope this helps.

Enjoy your day!

Cheers,
/Charbel

About Charbel Nemnom 271 Articles
Charbel Nemnom is a Microsoft Cloud Consultant and Technical Evangelist, totally fan of the latest's IT platform solutions, accomplished hands-on technical professional with over 15 years of broad IT Infrastructure experience serving on and guiding technical teams to optimize performance of mission-critical enterprise systems. Excellent communicator adept at identifying business needs and bridging the gap between functional groups and technology to foster targeted and innovative IT project development. Well respected by peers through demonstrating passion for technology and performance improvement. Extensive practical knowledge of complex systems builds, network design and virtualization.

Be the first to comment

Leave a Reply