Microsoft’s NIC Teaming Black Belt #WS2012 #WS2012R2 #WS2016 #HyperV

Today, a blog reader commented on one of my previous post about the Converged Network Fabric in VMM

The comment was:

[LACP vs. Switch Independent, I know that Microsoft recommend Switch Independent / Dynamic mode, however I can’t find what their pros and cons are. Network administrators here always prefer using LACP].

I had a discussion recently about the same topic with one of my colleague at work, I thought this is the right time to talk about NIC teaming options in the Host and Switch(es) with LACP and Switch Independent mode. The NIC teaming in general is a confusing topic for some people. 

In today’s blog post, I will deep dive into Microsoft NIC Teaming options starting from Windows Server 2012, 2012 R2 and what’s coming in Windows Server 2016, I always hear people saying that Microsoft recommends Switch Independent / Dynamic mode in all cases, and why people are still using LACP, the answer is they are cases where one is a little better than the other and has more options that the other don’t, I will address that by the end of this post.

NICTeaming-11

So without further ado, let’s start from the basics and then move into the advanced topics.

What is NIC Teaming?

NIC Teaming is also referred as NIC Bonding called by some people, Load Balancing and Failover (LBFO).

In short, the combining of two or more network adapters so that the software above the team perceives them as a single adapter as one pipe that incorporates failure protection and bandwidth aggregation.

NIC teaming solutions historically have also provide per-VLAN interfaces for VLAN traffic segregation, and Microsoft teaming solution of course does the same thing, I will get to this shortly.

Why use Microsoft’s NIC Teaming?

  • Vendor agnostic – anyone’s NICs can be added to the team.
  • Fully integrated with Windows Server 2012 / 2012 R2 / 2016.
  • Let’s you configure your teams to meet your needs.
  • Server Manager-style UI that manages multiple servers at a time.
  • Microsoft fully supported! so no more calls to NIC vendors for teaming support or getting told to turn off teaming.
  • Many vendors on the market has dropped down the teaming business.
  • Team management is easy using PowerShell, System Center Virtual Machine Manager and Server Manager.

Microsoft’s NIC Teaming Vocabulary Lesson

    • pNICs = Physical NICs on the host, or Network Adapters.
    • tNICs = Team Interfaces exposed by the team, Team NICs, or tNICs.
    • vNICs = Interfaces exposed by the Hyper-V Virtual Switch to the Management OS (host partition).
    • vmNICs = Interfaces exposed by the Hyper-V Virtual Switch into a Virtual Machine.

4 Terms to be added to your dictionary Smile (pNICs, tNICs, vNICs and vmNICs).

Teaming Modes / Load Distribution Methods

NICTeaming-04

Teaming Modes

  • Switch independent mode
    • The host is not dependent on the switch (No configuration is required on the physical switch).
    • Protects against adjacent switch failures (do not cause the team to break).
    • Easy to configure and it’s the easiest one to not mess up when you deploys it.

NICTeaming-02

  • Switch dependent modes
    • Static Teaming (or Generic teaming) which is switch dependent teaming without protocol, there is not a lot of value in using Switch dependent teaming without the protocol.
    • LACP (802.1ax teaming, also known as 802.3ad). The 802.1ax is IEEE standard that defines the Link Aggregation control protocol.
    • Requires configuration of the adjacent switch (very dependent on configuration of the switch).
    • If you are looking at switch dependent mode, you should really look at LACP (I will get to Why LACP shortly).

NICTeaming-03

Load Distribution Modes

  • Address Hash – comes in 3 flavors
    • 4-tuple hash: (Default distribution mode) uses the RSS hash if available, otherwise hashes the TCP/UDP ports and the IP addresses. If ports not available, uses 2 tuple instead.
    • 2-tuple hash: hashes the IP addresses. If not IP traffic uses MAC-address hash instead.
    • MAC address hash: hashes the MAC addresses.
    • All of the traffic for the host arrives on one NIC, this is not very useful in a Hyper-V case, but quite useful in a native teaming case, because in a native teaming case you generally have only one MAC address visible to the network from the tNIC anyway.   
  • Hyper-V Port
    • Hashes on the port number of the Hyper-V switch that the traffic is coming from (All traffic from a given VM to a given team member only, and of course when you have too many VMs, then multiple VMs will be mapped to each team member).
    • Recommended to use with Hyper-V 2012.
  • Dynamic
    • Recommended to use with Hyper-V 2012 R2.
    • Dynamic distribution is Address Hash on the outbound side, and Hyper-V Port on the inbound side (are you confused yet? probably Confused smile).
    • What the means is that the outgoing traffic will be spread across the team members on per flow basis, and then watch the ARP and manage the ARPs coming from the VMs (ARP responses) in a way that ensures that each of the VM has their incoming traffic mapped to various different team members, so if you have a lot of team members for example you have a team of 8X1Gig NICs, this means that will take the VMs and distribute them across all incoming team members, although each VM will be mapped to exactly one NIC per incoming purposes, that means that a given VM traffic cannot exceed the bandwidth of a single team member. However on the outbound side, the distribution is actually on per flow basis, so a given VM can generate more than 1 team member worth of traffic, and will break down into flows and distribute them across the team members.
    • And because Dynamic is based on Flowlet technology, Microsoft keep checking gaps in the flows, and after each gap has occurred in the flow, they look whether the flow should continue on the same NIC or whether there is a less used NIC that can move that flow to, in order to balance that outbound traffic across all of the NICs. 

NICTeaming-14

Teaming Modes and Load Distribution Methods (Summary)

NICTeaming-05

Active/Standby

  • A frequently used mode with NO real value.
  • Available only in Switch Independent / Address Hash operation.
  • Only one team member can be set to standby.

NICTeaming-15

I like to give the analogy of building a 4 lanes highway, that’s a free way with two lanes in each direction, and then taking one lane in each direction out of service till the other lane is broken.

   NICTeaming-07

You already have the infrastructure investment made that your company paid for, you have already got all of the connections and everything in place, and you are not using half of it because you want to be there in case you need it when the other one brakes.

It makes a lot better sense to use Active/Active, such that you are always using all of the infrastructure that you already bought paid for.

Windows Server 2012 Switch / Load Interactions

 NICTeaming-06

Windows Server 2012 R2 Switch / Load Interactions

NICTeaming-13

Team Interfaces (tNICs)

  • Team interfaces can be in one of two modes:
    • Default mode: passes all traffic that doesn’t match any another team interface’s VLAN id.
    • VLAN mode: passes all traffic that matches the VLAN.
  • Inbound traffic is always passed to at most one team interface only.

NICTeaming-08

The Hyper-V team have said loud and clear, if you are using Hyper-V Virtual Switch on top of a team, the team must only exposed the default interface (interface a default mode) and no others. The Hyper-V virtual switch manage all of the VLANs configuration, it’s perfectly capable of that.   

  • Team interfaces created after initial team creation must be VLAN mode team interfaces.
  • Team interfaces created after initial team creation can be deleted at any time (using server manager UI or PowerShell). The primary interface cannot be deleted except by deleting the team.
  • It is a violation of Hyper-V rules to have more than one team interface on a team that is bound to the Hyper-V Switch.
  • A team with only one member (one pNIC) may be created for the purpose of disambiguating VLANs.
  • A team of one has no protection against failure (of course Winking smile).

Frequently Asked Questions

  • Any physical Ethernet adapter can be a team member and will work as long as the NIC meets the Windows Logo requirements.
  • Teaming of RDMA adapters is not supported in Windows Server 2012 and 2012 R2, but supported in Windows Server 2016 (I’ll get to this shortly).
  • Teaming of WiFi, WWAN, etc, adapters is not supported.
  • Teams of teams are not supported as well.
  • Teaming in a Virtual Machine is supported, but limited to:
    • Switch Independent, Address Hash mode only.
    • Teams of two team members are supported.
    • Intended/optimized to support teaming of SR-IOV Virtual Functions (VFs) but may be used with any interfaces in the VM.
    • Requires configuration of the Hyper-V Virtual or failovers may cause loss of connectivity.

NICTeaming-09

  • Maximum number of NICs in a team: 32
  • Maximum number of team interfaces: 32
  • Maximum teams in a server: 32
  • Not all maximums may be available at the same time due to other systems constraints.

NIC Teaming Manageability

  • Easy-to-use NIC Teaming UI with Server Manager (lbfoadmin.exe)
    • Intuitive and Powerful.
    • UI operates completely through PowerShell – uses PowerShell cmdlets for all operations.
    • Manages Servers (including Server Core) remotely from Windows 8, 8.1 and 10 clients.
  • Powerful PowerShell cmdlets
    • Object: NetlbfoTeam (New, Get, Set, Rename, Remove).
    • Object: NetLbfoTeamNic (Add, Get, Set, Remove).
    • Object: NetlbfoTeamMember (Add, Get, Set, Remove).
  • System Center Virtual Machine Manager
    • Intuitive and Powerful for large deployment.
    • Deployment through predefined templates and profiles.
    • Consistent deployment across all hosts.

Which NIC Teaming Mode Shall I Choose?

  • Switch Independent
    • Doesn’t depended on switch configuration.
    • Balances outbound traffic especially in Dynamic distribution mode.
    • Maps inbound traffic to different team members based on per vPort basis
      • Limits inbound traffic to a given vPort to the bandwidth of a single team member (more of an issue in 1G interfaces than 10G, 40G or 100G interfaces).
  • Switch Dependent (LACP)
    • All interfaces in Link Aggregation Group (LAG) treated as single pipe except:
      • Packets in the same flow are placed on the same team member to reduce opportunity for misordering.
    • Host manages outbound packet placement, Switch determines inbound packet placement.
    • LAGs and MLAGs:
      • Many people are using multi-chassis switches (a.k.a Stacked Switches).
      • Link Aggregation Group (LAG) Versus Multi-chassis Link Aggregation Group (MLAG).
      • Every Switch vendor does it differently and depending on what vendors are you using, you may have better or bad results.  

NICTeaming-10

Advantages and Disadvantages (Switch Independent vs. LACP)

  • Switch Independent
    • The main one is (No switch configuration is required).
      • Less opportunity for misconfiguration
    • Does a good job of load spreading.
    • Adequate for vast majority of deployments.
    • Works with Windows Server 2016 RDMA teaming since host determines which interface traffic arrives on.
    • Detect cable faults, NIC faults, adjacent switch power issues, etc. but doesn’t detect dead switch port logic (This is extremely rare failure, this is the case where a switch is still electrically alive, but the logic on the port has hung or failed. The switch independent is looking at the electric interface and won’t detect that the switch has quick passing traffic on a given port).   
  • Why LACP
    • Because it maintains heartbeat between the switch port logic and the host, this heartbeat allows to detect that switch port logic errors or failure, because if the switch port logic goes down, it will not send any heartbeat, it does not process the heartbeat, it does not send back the response, and the result is that NIC teaming on the host will detect that the switch port is not alive anymore, then they will take that particular link out of the LAG for the duration time the switch port is not responding.      .
    • Allows switch to load balance ingress flows across the team members.
      • Integrate well with Equal-cost multi-path (ECMP) through Multi-chassis switches.
    • Does not work with Windows Server 2016 (RDMA) teaming, because stateful offloads like RDMA requires all the traffic for that engine to arrive on a given NIC, so LACP won’t work.

What’s New for NIC Teaming in Windows Server 2016 Technical Preview 4?

  • Switch-embedded Teaming (SET) where the teaming will be integrated into the Hyper-V Virtual Switch.
  • SET is an alternative NIC Teaming solution that you can use in environments that include Hyper-V and the Software Defined Networking (SDN) stack in Windows Server 2016 Technical Preview.
  • The Switch-embedded Teaming (SET) mode will support RDMA and SR-IOV teaming.
  • Supporting SDN-switch capabilities (Packet Direct, Converged vNIC and SDN Quality of Service).
  • The Switch-embedded Teaming (SET) will be limited to:
    • Switch Independent teaming mode only.
    • Dynamic and Hyper-V Port mode only for load distribution.
    • Managed by SCVMM or PowerShell only, but not with Server Manager or (lbfoadmin.exe).
    • Only teams identical ports (same manufacturer, same model, same firmware and driver).
    • The Switch must be created in SET-mode. (you cannot change it later).
    • Up to eight physical Ethernet network adapters into one or more software-based virtual network adapters.
    • The use of SET is only supported in Hyper-V Virtual Switch in Windows Server 2016 Technical Preview. You cannot deploy SET in Windows Server 2012 R2.
  • In another article, I’ll demonstrate how to configure and use Switch-embedded Teaming (SET) in Windows Server 2016 TP4.

For further reading, you can download the guides from TechNet Gallery in Word format at the following locations:

Congratulations! You have completed Microsoft® Black Belt NIC Teaming Certificate Winking smile

NICTeaming-12

Thanks for reading!

Cheers,
-Charbel

About Charbel Nemnom 313 Articles
Charbel Nemnom is a Microsoft Cloud Consultant and Technical Evangelist, totally fan of the latest's IT platform solutions, accomplished hands-on technical professional with over 15 years of broad IT Infrastructure experience serving on and guiding technical teams to optimize performance of mission-critical enterprise systems. Excellent communicator adept at identifying business needs and bridging the gap between functional groups and technology to foster targeted and innovative IT project development. Well respected by peers through demonstrating passion for technology and performance improvement. Extensive practical knowledge of complex systems builds, network design and virtualization.

Be the first to comment

Leave a Reply