Sometimes things go wrong with Hyper-V Replica. There are many reasons for this, but one of the reason is when you are running out of disk space on the replica side.
Are you monitoring your Hyper-V Replica infrastructure? If not, then make sure to check this post <Advanced Hyper-V Replica Monitoring using PowerShell>.
In today’s blog post, I will show you how to remove lingering replica log file from a Hyper-V Virtual Machine that were not properly removed after removing VM Replication on Windows Server 2012 R2 Hyper-V. The same will apply whether you are using Windows Server 2016 or later.
A little bit of context on how Hyper-V replica works behind the scene, Hyper-V replica creates checkpoint (.avhdx) to handle the initial replication of the virtual machine (for example when getting the first copy of the virtual machine across the replica host). Hyper-V Replica will merge this disk out once it has successfully made a full copy of the virtual machine, virtual hard disk (which can take a while).
As you can see in above screenshot, the replication is not enabled for this virtual machine, in fact I removed all replication using Remove-VMReplication cmdlet when the replication health was in critical state.
If we look at the virtual hard disk files of the VM, we see the traces of a Windows Server 2012R2 Hyper-V replica (HRL Hyper-V replica log).
Did you notice the HRL file size is large 32GB.
A quick background about HRL, once the initial replication of a VM is completed, Hyper-V Replica will start monitoring the changes of each selected virtual hard disk. This is done by mirroring the changes to a Hyper-V Replica Log (HRL) file. The HRL files are kept in the same directory where Virtual Machine VHD files reside. Each VHD file, which has been selected for the replication, has an associated HRL file. The changes are tracked at the Virtual Machine level e.g. any changes (write operations) to the virtual machine are tracked and stored in the Hyper-V Replication Log files. Once the replicas/HRL files are ready, “Replication Manager” notifies “Replication Tracker” (RT) to send the copies to the Replica Server. The changes are tracked on the Primary Server and Change Tracking component has no function on the Replica Server!
The question is why the size of the log file (.HRL) is growing from MBs to GBs? if you want to identify the guest processes that cause large log files, then make sure to download (Hyper-V Replica Identify Script) from TechNet and run it inside the guest OS, this script provides debugging information that will help you to identify what is going on in the guest that causes large log files in Hyper-V Replica.
As Best Practice, please make sure to create a new VHD(x) file, one for each VM, and move the paging files to those volumes. when you set up Hyper-V replication, do NOT select the box next to that volume.
Also, if you have any VM that hosts SQL server, then move the TempDB files to a separate VHD(x) file as well, and don’t replicate that one, because every time you start a new server or new SQL instance, the paging files and Temp DBs are empty, so nothing is lost if you fail over to the replica side.
If you look at all the remaining HRL files on the host, we can see around 81GB of disk space.
You can delete each HRL log file manually as long as the VM replication is removed.
PowerShell to the rescue!
Get-ChildItem -File -Path D:\ -Filter *.HRL -Recurse | Remove-Item –Force
As a side note: If you are backing up virtual machines running on Windows Server 2016 Hyper-V or later, you will see two new files, MRT (Modified Region Table) file, and RCT (Resilient Change Tracking) which keep track for all the changed blocks and travel with each (VHDX/AVHDX) if a virtual machine is moved to another Hyper-V host or to another storage. If you come across a scenario where you want to delete the MRT and RCT files, then you can use the same script described above to remove unused MRT/CRT files as well. However, in normal situation, you should not touch those files as long as the virtual machine is protected by a backup application such as (DPM, VEEAM, etc.). The size of the MRT and RCT files is very small less than < 100 KB.
Do you want to know more about RCT and MRT files in Windows Server 2016 Hyper-V including backup architecture and the difference between different versions? I strongly recommend to check my recent published book Windows Server 2016 Hyper-V Cookbook – Second Edition!
Keep monitoring your Hyper-V Replica / Backup and start removing the lingering HRL log files and free up some disk space.