Background disk merge failed to complete: The requested operation could not be completed due to a file system limitation (0x80070299) #HyperV

[Updated: March 4th, 2016] – Adding a second recovery option under Resolution section.

Issue

Sometimes things go wrong with Hyper-V replica or backup jobs, especially when Hyper-V replica and backup are running at the same time. In fact, this is the reason of this blog post.

One of my colleague was backing up his virtual machine, the VM was in backup state, and then he initiated replication from Host A to Host B at the same time.

The Virtual Machine ended-up by having two (.avhdx) as shown in the following screenshot: HyperV-BackgroundMergeFailed-01

Overview

A little bit of context on how Hyper-V replica and backup works behind the scene in Windows Server 2012 R2 Hyper-V.

I mentioned Windows Server 2012 R2 here, because there are some changes coming in Windows Server 2016.

Hyper-V replica creates checkpoint (.avhdx) to handle the initial replication of the virtual machine (for example when getting the first copy of the virtual machine across the replica host). Hyper-V Replica will merge this disk out once it has successfully made a full copy of the virtual machine, virtual hard disk (which can take a while).

Hyper-V backup also creates checkpoint (.avhdx). In fact, a checkpoint is created for every virtual machine that is to be backed up. The I/O writes are temporarily redirected to the checkpoint’s AVHDX/AHVD file(s). This gives the backup requestor a clean & crash consistent copy of the virtual machine’s VHD(X) files that are safe to read. After the backup, the checkpoint is merged and the job is completed.

Symptoms

After the backup is completed, the snapshot is being merged automatically, But since Hyper-V Replica and backup were running at the same time. The background disk merge failed to complete.

The Microsoft-Windows-Hyper-V-VMMS/Admin event log will be filled with the Error ID: 19100, because Hyper-V will continuous try to run the disk merge in the background and it will fail.

HyperV-BackgroundMergeFailed-03

If you look at the checkpoints of this Virtual Machine in Hyper-V Manager or using PowerShell, you don’t find any!

HyperV-BackgroundMergeFailed-04

HyperV-BackgroundMergeFailed-05

However if you open the VM settings for this virtual machine, you can see that Edit disk is not available because checkpoints exist for this virtual machine.

HyperV-BackgroundMergeFailed-02

Resolution

We have actually two options to resolve this issue, but before we start, you need to remove first the replication for the affected VM on the primary host, and you want to stop the backup job if any is still running.

Option 1

Option 1 is risky, because we need to start merging those checkpoints manually to the parent VHD(X), hence if you missed any step, you may loose your data!

In between, there is no need to dive in and start merging the checkpoints manually unless you are in a such scenario where that’s the only option you have left, for example like this case and you don’t have another Hyper-V host.

You can use PowerShell or DiskPart for this but I see a value by using Hyper-V Manager UI here, because you can inspect which the most recent (.avhdx) and it’s parent, and then proceed safely.

It’s very important to start with the most recent virtual hard disk and then work your way down.

My fellow MVP, Didier Van Hoye wrote a great blog post on how to do this operation manually.

Walkthrough:

First we need to find the parents of all (.avhdx) files.

Open Hyper-V Manager and click on Inspect Disk…

HyperV-BackgroundMergeFailed-06

After you inspect the checkpoints and their parents, you can start Editing the disk.

As a side note, I strongly recommend to take a note for every step here.

Open Hyper-V Manager and click on Edit Disk… Browse and select the most recent (.avhdx) and click Next>

In my case it is (SQL-Instance_CA3AD8B3-C205-48A7-930B-24D8F7375334.avhdx)

HyperV-BackgroundMergeFailed-07

At the Edit Virtual Hard Disk Wizard, select Merge and then click Next>

HyperV-BackgroundMergeFailed-08

In my case, I want to merge to it’s parent disk which is (SQL-Instance_4B2591D7-440C-4A12-836B-64D055D38E5D.avhdx)

HyperV-BackgroundMergeFailed-09

Verify and confirm the merge options are correct and click Finish

HyperV-BackgroundMergeFailed-10

Let the wizard complete the merge.

Now if you go back and look at the virtual machine settings, you can see that the most recent (.avhdx) file that you just merged is still there. However, this file does not exist anymore.

HyperV-BackgroundMergeFailed-02

Because the virtual machine was unable to auto-merge it’s disk in the background, you need to manually attach the most recent (.avhdx).

To do that, you need to click Browse, and select the new parent disk, in my case it is (SQL-Instance_4B2591D7-440C-4A12-836B-64D055D38E5D.avhdx).

HyperV-BackgroundMergeFailed-11

When you hit apply, you will receive the following Warning Data loss may occur!

HyperV-BackgroundMergeFailed-12

Click Continue. You will receive a final warning message!

HyperV-BackgroundMergeFailed-13

This is not safe, easy and obvious so you are responsible for everything you do.

You continue do this over and over again until you merge the last (.avhdx) into the parent VHDX.

HyperV-BackgroundMergeFailed-14

One thing worth point out is that you may encounter the same error during the manual merging process, one of the disks might fail to merge to it’s parent directly.

HyperV-BackgroundMergeFailed-15

In this case you need to merge the disk to a new VHD(X).

HyperV-BackgroundMergeFailed-16

Option 2

Option 2 is less risky if this option is available to you versus error-prone using option 1.

This option requires to have a second Hyper-V host in your environment, and I believe we do all have more than one host in production.

We need to leverage Shared-Nothing live migration feature and move the affected virtual machine including it’s storage to the second Hyper-V host.

You can use PowerShell or Hyper-V Manager, it’s much easier with PowerShell.

You might ask why we don’t use storage migration and move the storage within the first host. If you recall, the primary host kept on trying to run disk merge in the background and was failing, thus this option is not viable for us. However, by moving the VM to a different host, and when the migration is completed, the second host will trigger a new disk merge and merge all the disks for you.

Note: As a measure of precaution, I strongly recommend doing this process while the VM is in off state, thus will avoid BSOD screen or reboot even if the VM is Generation 2.

Takeaways

Please do not run backup and Hyper-V replica for your virtual machines at the same time.

The changes introduced in Windows Server 2012 R2 would benefit using any backup product to take backup of Replica VMs, in other words, if you are using Hyper-V replica today, you could take the benefit of that and start backing up your replica VMs. There are a few key scenarios where backup of a Replica VM becomes useful:

1- Reduce the impact of backup on the running workload (primary Host and VMs).
2- If you have Limited bandwidth between sites.
3- If you are hosting your virtual machines in a Hoster datacenter as the replica site.

Please remember to have backup all the time before you start this process.

Thanks for reading and hope this helps someone!

Cheers,
-Charbel

About Charbel Nemnom 325 Articles

Charbel Nemnom is a Microsoft Cloud Consultant and Technical Evangelist, totally fan of the latest’s IT platform solutions, accomplished hands-on technical professional with over 15 years of broad IT Infrastructure experience serving on and guiding technical teams to optimize performance of mission-critical enterprise systems. Excellent communicator adept at identifying business needs and bridging the gap between functional groups and technology to foster targeted and innovative IT project development. Well respected by peers through demonstrating passion for technology and performance improvement. Extensive practical knowledge of complex systems builds, network design and virtualization.

Be the first to comment

Leave a Reply