In this blog post, we’ll look at how to isolate Hyper-V Replica Broker in a cluster over a dedicated network.
Table of Contents
Introduction
High Available (HA) is not a solution for Disaster Recovery (DR) and cannot replace DR. Actually HA is a component of a DR plan, which consists of processes, policies, procedures, backup and recovery plan, documentation, tests, best practices, and so on. The objective of a Disaster Recovery Plan is simply to have business continuity in case of any disaster.
In Windows Server 2012 and beyond, Microsoft addressed this challenge by offering an amazing solution into the platform called Hyper-V Replica making DR replication possible and affordable for all sized companies.
Hyper-V Replica introduces the ability to replicate a virtual machine asynchronously to a secondary Hyper-V host, such as a remote disaster recovery site, using a single network connection.
Hyper-V Replica is also applicable to a failover cluster. With a failover cluster of Hyper-V hosts in place, we need first to configure a Hyper-V Replica Broker role using Failover Cluster Manager.
Ordinarily, Hyper-V Replica can be enabled in the Hyper-V Settings of a Hyper-V Host. However, in a failover cluster, the Hyper-V Replica is grayed out in an individual Hyper-V host’s Hyper-V Settings. Instead, all Hyper-V Replica settings are to be managed through the Hyper-V Replica Broker.
Hyper-V Replica (Broker) isolated network for Hyper-V is not listed as a requirement by Microsoft. However, you can throttle Hyper-V Replica Traffic by using the New-NetQosPolicy cmdlet, but I strongly recommend isolating the replica traffic from the host Management OS to a dedicated network, and by leveraging the converged network in Hyper-V were combining multiple physical NICs with NIC teaming and QoS we can isolate each network traffic while maintaining resiliency.
In this blog post, we’ll look at how to isolate Hyper-V Replica in a cluster over a dedicated network.
Hyper-V Replica Cluster
To configure Hyper-V Replica Broker over a dedicated and isolated replication network, there are a few things you want to take into consideration.
1) Ensure you have an adequate teamed network or converged fabric with vNIC dedicated to the Hyper-V Replica network on each Hyper-V node. In this instance, I have a dedicated vNIC for Hyper-V Replica Network.
2) Before you install the Hyper-V Replica Broker Role. The Network in the cluster that will be used for Hyper-V Replica must be configured to ‘Allow clients to connect through this network’ in Failover Cluster Manager, Networks, as shown in the following screenshot:
3) In the Failover Cluster Manager, in the left pane, and while the cluster name is highlighted, click Configure Role in the Actions pane. The High Availability wizard opens. In the Select Role screen, select Hyper-V Replica Broker as shown in the following screenshot:
4) Complete the wizard, providing a NetBIOS name to be used as the connection point to the cluster (called a “client access point”), and the IP address that will be used for the Replica network.
5) Verify that the Hyper-V Replica Broker role comes online successfully and can failover between all nodes of the cluster. To do this, right-click the role, point to Move, and then click Select Node. Then, select a node, and then click OK.
6) Configure your Hosts file to include the FQDN and NetBIOS names of each Hyper-V host that needs to be replicated “to and from”. In this scenario, you can have a standalone host and a cluster working as a primary or replica server, or a cluster for each.
Hyper-V Replica (Broker) will still use DNS to resolve the names of each Hyper-V host and as a result, will still resolve the Parent Management OS IP Address. Do you want to influence and let Microsoft change the Hyper-V replication from DNS to IP-based? if so, please vote at the Windows Server user voice page here, and let your voice shape the product development. Your vote is very important. At the time of writing, the only option is to update the C:\windows\system32\drivers\etc\hosts file to include the replica network IP address on EVERY SINGLE HYPER-V Host. This will need to be configured identically on the corresponding Replica Site as well.
7) Open Failover Cluster Manager, right-click on the Hyper-V Replica Broker and select the Replication Settings action. Enable replication on HTTP and/or HTTPS, depending on the trust level of the replication network based in your environment. In the Authorization and storage section, as shown in the following screenshot, use the radio buttons to specify whether to allow any authenticated (primary) server to send replication data to this Replica server or to limit acceptance to data from specific primary servers. If you specify individual primary servers, you can designate a separate storage location for Replica data for each one, as well as group them with the Trust Group tag.
8) Ensure the inbound TCP exception for port ‘80’ or ‘443’ is enabled in the firewall on every host that will be used as a member of the Hyper-V Replica. If you are using Windows Firewall, enable “Hyper-V Replica HTTPS Listener (TCP-In)” and “Hyper-V Replica HTTPS Listener (TCP-In)” rules.
9) If you need to route the replica traffic to another site, you will need to configure a persistent route to ensure Hyper-V Replica traffic will route correctly. You need to use the route command (route add) to configure a static route on all Hyper-V hosts. Open a command prompt and type the following (update according to your environment): route add 10.8.1.0 mask 255.255.255.0 172.16.20.1
If you want to remove a static route, you would type (route delete). To list all the routes, you would type (route print).
10) Once the Hyper-V Replica Cluster is configured on a dedicated network, you can enable replication for each virtual machine that you want to be replicated. The details of replication are configured on a per-virtual-machine basis, so each one can have slightly different settings.
11) Last but not least, check Task Manager Network performance to ensure that the Initial Replication is copying across the dedicated Hyper-V Replica Cluster Network.
Conclusion
Keeping the Hyper-V Replica traffic off your production network allows you to segregate and utilize more bandwidth without sharing it with the management OS network. However, this does not mean that you will not see potential performance issues as the servers are replicating, but the impact will be much less especially when you isolate the replica traffic and throttle the network bandwidth of your Hyper-V hosts as described in this post.
As always, these tips and tricks are meant for guidance and process, actual results will vary based on your hardware and circuit types.
Learn more
Do you want to learn about Windows Server Hyper-V, I highly encourage you to check the Windows Server Hyper-V Cookbook for in-depth details about Hyper-V and automation tasks.
__
Thank you for reading my blog.
If you have any questions or feedback, please leave a comment.
-Charbel Nemnom-
Where do i find step 2? On the NIC, In hyper-v manager? It’s not in failover cluster manager because it’s not supposed to be installed yet. Also I’m using windows 2016. Is it the same?
Hello Rich, step 2 is located in Failover Cluster Manager (Networks) section.
I have tried this over and over the traffic still goes over the management network. When I create the replica broker it auto assigns a DHCP IP from my production network as well as the 192.168.101.0 network which I specify. I delete the DHCP IP from the broker so just the replication network address is there still no joy. host files have been changed on all nodes.
Hello Ken, thanks for the comment!
I highly recommend when creating the replica broker, please set the static IP address during the creation process and not afterward.
The next thing that you can check is to verify the routing table on each host as described in Step 9 in this article (route print).
You can also try to force it with (route add).
Hope it helps!
I got 2 Node S2D clusters and Hyper-V Replica broker configured,
outgoing replications from cluster to a workgroup Hyper-V host works fine,
but I can’t get incoming replication to work at all.
whenever I try to set up replication from an external workgroup server to a cluster replica broker, I keep getting the below error.
Hyper-V failed to enable replication for virtual machine ‘ABC’: The connection with the server was terminated abnormally (0x00002EFE). (Virtual machine ID E59616A4-4EA0-44B1-8455-0812898171D4)
Hyper-V failed to establish a connection with the Replica server ‘s2dcl1n1.xxxxx.co.uk’ on port ‘443’. Error: The connection with the server was terminated abnormally (0x00002EFE).
I have SSL certificates installed on all hosts. and workgroup host can do replication to other hosts but not cluster host.
not sure if something is missing?
Harry
Hello Harry, thanks for the comment!
How did you create the Self Signed Certificates?
I assume that your firewall rules are set correctly.
Please make sure to open port 443 from the Physical IP address of Each Hyper-V Node which is part of the Cluster to the IP of the Hyper-V external workgroup at the DR Site.
Hope it helps!
Hi Charbel,
I am using an SSL certificate from a provider so not self-signed, it’s *.domain.com I use it for all other hosts.
All firewalls are OFF.
I got it working now by changing the SSL port from 443 to 4434 on hyper-v Replica broker.
not sure if 443 is used by something else on the cluster but it just won’t work on 443.
all working now thanks
Harry
Hello Harry, good to hear that the issue is resolved now.
I suspected that the issue lies on the networking side.
Is the cluster located in your data center or under the provider’s control?
You can run netstat -a -n and check which service is using port 443.
Hi,
Its Windows Admin Center that’s running and using port 443 on Cluster nodes.
wish Replication settings can detect and complain if the port is not free etc.
anyway all working for me by changing port so all good thanks.
Harry
Here you go, you are running Windows Admin Center on the cluster occupying the default port 443.
Great all is sorted out now!
Hello Sir, Great Article.. Is there any option to enable Group of VMs replication, Failover, Test failover etc..? It will be excellent if we have GUI replication webpage with above features.
Hello Sandeep, thank you for the feedback and comment!
Yes, you can do that, look at the following article: VM Groups and VM Start Ordering in Windows Server Hyper-V.
However, this is done only using PowerShell, there is no GUI to replicate Group of VMs.
Hope it helps!
Thank you for the response. VM grouping is good solution. We had full Azure ASR between sites, but MS deprecated the features. No other vendors are offering similar solutions like test failover, GUI based replications and map a particular test network during test failover between sites.
Hello Sandeep, thank you for providing more context here.
Yes, I have seen the deprecation announcement of disaster recovery between customer-managed sites (with VMM) using Azure Site Recovery as of March 2023.
I see Microsoft are directing customers to start using Azure as the DR target and not site-to-site replication.
However, you can choose to continue with site-to-site replication using the underlying Hyper-V Replica native solution. But you have to use mixed of tools like PowerShell, or Hyper-V Manager.
You can even go a step further and use System Center 2022 Orchestrator (Runbooks) to orchestrate and automate site-to-site Hyper-V replication.
Hope it helps!