NIC Teaming

Prévia do material em texto

8
Using the multiple NICs of your File Server running Windows
Server 2008 (and 2008 R2)
Jose Barreto - MSFT 3 Sep 2010 8:07 AM
1 - Overview
 
When you set up a File Server, there are advantages to configuring multiple Network Interface Cards (NICs). However,
there are many options to consider depending on how your network and services are laid out. Since networking (along
with storage) is one of the most common bottlenecks in a file server deployment, this is a topic worth investigating.
Throughout this blog post, we will look into different configurations for Windows Server 2008 (and 2008 R2) where a file
server uses multiple NICs. Next, we’ll describe how the behavior of the SMB client can help distribute the load for a file
server with multiple NICs. We will also discuss SMB2 Durability and how it can recover from certain network failure in
configuration where multiple network paths between clients and servers are available. Finally, we will look closely into the
configuration of a Clustered File Server with multiple client-facing NICs.
 
2 – Configurations
 
We'll start by examining 8 distinct configurations where a file server has multiple NICs. These are by no means the only
possible configurations, but each one has a unique characteristic that is used to introduce a concept on this subject.
 
2.1 – Standalone File Server, 2 NICs on server, one disabled
 
This first configuration shows the sad state of many File Servers out there. There are multiple network interfaces available,
but only one is actively being used. The other is not connected and possibly disabled. Most server hardware these days
does include at least two 1GbE interfaces, but sometimes the deployment planning did not include the additional cabling
and configuration to use it. Ironically, a single 1GbE (which provides roughly 100Mbytes per second of throughput) is a
common bottleneck for your file server, especially when reading data from cache or from many disk spindles (physical disk
throughput is the other most common bottleneck).
Having a single NIC has an additional performance downside if that NIC does not support Receive-side Scaling (RSS).
When RSS is not available, a single CPU services all the interrupts from a network adapter. For instance, If you have an
8-core file server using a single non-RSS NIC, that NIC will affinitize to one of the 8 cores, making it even more likely to
become a bottleneck. To learn more about deploying RSS, check http://www.microsoft.com/whdc/device/network
/NDIS_RSS.mspx.
 
2.2 – Standalone File Server, 2 NICs on server, teamed
 
A blog by Jose Barreto, a member of the File Server team at Microsoft.
All About
Windows Server
Cloud OS Blogs Datacenter
Management
Client
Management
Virtualization,
VDI & Remote
Desktop
File & Storage &
High Availability
Windows Server
Management
Identity & Access
Jose Barreto's Blog
Using the multiple NICs of your File Server running Windows Server ... http://blogs.technet.com/b/josebda/archive/2010/09/03/using-the-multi...
1 de 12 27/08/2014 07:33
One simple and effective solution for enabling the multiple NICs on a File Server is NIC Teaming, also known as “Link
Aggregation” or “Load Balancing and Failover (LBFO)”. These solutions, providers by vendors like Intel, Broadcom and HP,
effectively combine multiple physical NICs into one logical or virtual NIC. The details vary based on the specific solution,
but most will provide an increase of throughput and also tolerance to the failure of a NIC or to a network cable being
accidentally unplugged.
The NIC team typically behaves as a single NIC, requiring only a single IP address. Once you configure the team itself, the
Windows Server and File Server configuration proceeds as if you had only one NIC. However, NIC teaming is not
something included with Windows Server 2008 or Windows Server 2008 R2. Support for these solutions (the hardware, the
drivers and the configuration tools) is provided by the hardware manufacturer.
You can find Microsoft’s support policy for these types of solutions at http://support.microsoft.com/kb/254101 and
http://support.microsoft.com/kb/968703.
 
2.3 – Standalone File Server, 2 NICs on server, single subnet
 
If you don’t have a NIC teaming solution available but you are configuring a File Server, there are still ways to make it
work. You can simply enable and configure both NICs, which will each need their own IP address. If everything is
configured properly, both IP addresses will be published to DNS under the name of File Server. The SMB client will then be
able to query DNS for the file server name, find that it has multiple IP addresses and choose one of them. Due to DNS
round robin, chances are the clients will be spread across the NICs on the file server.
There are several Windows Server components contributing to make this work. First, there’s the fact the File Server will
listen on all configured network interfaces. Second, there’s the dynamic DNS that automatically registers all the IP
addresses under the server’s name (if configured properly). Third, there’s the fact that DNS will naturally round robin
through the different addresses registered under the same name. Last but not least, there is the File Client that will use
one of the available IP addresses, giving a priority to the first address on the list (to honor the DNS round robin) but using
one of the others if the first one does not respond quickly. The SMB client will use only one of the IP addresses at once.
More on this later.
What’s more, due to a feature called SMB2 durability, it’s possible that the SMB client will recover from the failure of a NIC
or network path even if it’s right in the middle of reading or writing a file. More on this later as well.
It's important to note that applications other the file server and file client might not behave properly with this
configuration (they might not listen on all interfaces, for instance). You might also run into issues updating routing tables,
especially in the case of a failure of a NIC or removal of a cable. These issues are documented in KB article 175767. For
these reasons, many will not recommend this specific setup with a single subnet.
2.4 – Standalone File Server, 2 NICs on server, multiple subnets
Using the multiple NICs of your File Server running Windows Server ... http://blogs.technet.com/b/josebda/archive/2010/09/03/using-the-multi...
2 de 12 27/08/2014 07:33
 
Another possible configuration is for each of the File Server NICs to connect to a different set of clients. This is useful to
give you additional overall throughput, since you get traffic coming into both NICs. However, in this case, you are using
different subnets. A typical case would be a small company where you have the first floor clients using one NIC and the
second floor using the other.
While both of the IP addresses get published to DNS (assuming everything is configured correctly) and each of the SMB
clients will learn of both, only one of them will be routable from a specific client. From the SMB client’s perspective, that is
fine. If one of them works, you will get connected. However, keep in mind that this configuration won’t give your clients a
dual path to the File Server. Each set of clients has only one way to get to the server. If a File Server NIC goes bad or if
someone unplugs one of the cables from the File Server, some of your clients will lose access while others will continue to
work fine.
 
2.5 – Standalone File Server, 2 NICs on server, multiple subnets, router
 
In larger networks, you will likely end up with various networks/subnets for both clients and servers, connected via a
router. At this point you probably did a whole lot of planning, your server subnets can easily be distinguished from your
client subnets and there’s a fair amount of redundancy, especially on the server side. A typical configuration on the server
side would include dual top-of-rackswitches on the server side, aggregated to a central switching/routing infrastructure.
If everything is configured properly, the File Servers will have two IP addresses each, both published to the dynamic DNS.
From a client perspective, you have a File Server name with multiple IP addresses. The clients here see something similar
to what clients see in configuration 2.3, except for the fact that the IP addresses for the client and servers are on different
subnets.
It is worth noting that, in this configuration and all the following ones, you could choose to leverage NIC teaming, as
described in configuration 2.2, if that is an option available from your hardware vendor. This configuration might bring
additional requirements, since each of the NICs in the team are going into a different switch. The configuration of
Windows Server itself would be simplified due to the single IP address, although additional teaming configuration with the
vendor tool will be added.
 
2.6 – Standalone File Server, 2 NICs on “clients” and servers, multiple subnets
 
This last standalone File Server configuration shows both clients and servers with 2 NICs each, using 2 distinct subnets.
While this configuration is unusual for regular Windows clients, servers are commonly configured this way for added
Using the multiple NICs of your File Server running Windows Server ... http://blogs.technet.com/b/josebda/archive/2010/09/03/using-the-multi...
3 de 12 27/08/2014 07:33
network fault tolerance. Here are a few examples of such server workloads:
IIS servers that store their HTM and JPG files on file share
SQL Servers that regularly send their database backups to a UNC path
Virtualization Servers that use the file server as a library server for ISO files
Remote Desktop Servers (Terminal Servers) where many users use the file server to store their home folders
SharePoint servers configured to crawl file shares and index them
You can imagine the computers below as part of the configuration 2.5 above, only with more servers to the right of the
router this time.
 
2.7 – Clustered File Servers, 2 NICs on servers, multiple subnets, router
 
If you are introducing dual network interfaces for fault tolerance, you are also likely interested in clustering. This
configuration takes config 2.6 and adds an extra file server to create a failover cluster for your file services. If you are
familiar with failover clustering you know that, in addition to the IP addresses required for the cluster nodes themselves,
you would need IP addresses for each cluster service (like File Server A and File Service B). More on this later.
Although we’re talking about Clustered File Services with Cluster IP addresses, the SMB clients will essentially see a File
Server name with multiple IP addresses for each clustered file service. In fact, the clients here see something similar to
configurations 2.3, 2.5 and 2.6. It’s worth noting that, if File Server 1 fails, Failover Clustering will move File Service A to
File Server 2, keeping the same IP addresses.
2.8 – Clustered File Server, 2 NICs on “clients” and servers, multiple subnets
 
This last configuration focuses on file clients that are servers themselves, as described in configuration 2.6. This time,
however, the File Servers are clustered. If you are interested in high availability for the file servers, it’s likely you would be
also clustering the other server, if the workload allows for it.
Using the multiple NICs of your File Server running Windows Server ... http://blogs.technet.com/b/josebda/archive/2010/09/03/using-the-multi...
4 de 12 27/08/2014 07:33
3 – Standalone File Server
 
3.1 – SMB Server and DNS
 
A Windows Server file server with multiple NICs enabled and configured for dynamic DNS will publish multiple IP
addresses to DNS for its name. In the example below, for instance, a server with 3 NICs each in a different subnet is shown.
You can see how this got published on DNS. For instance, FS1 shows with 3 DNS A records for 192.168.1.2, 192.168.2.2
and 192.168.3.2. It’s important to know also that the SMB server will listen on all interfaces by default.
 
3.2 – SMB Client and DNS
 
From an SMB client perspective, the computers will query DNS for the name of the File Server. In the case of the example
above, you will get an ordered list of IP addresses. You can query this information using the NSLOOKUP tool, as shown in
the screenshot below.
Using the multiple NICs of your File Server running Windows Server ... http://blogs.technet.com/b/josebda/archive/2010/09/03/using-the-multi...
5 de 12 27/08/2014 07:33
The SMB client will attempt to connect to all routable IP addresses offered. If more than one routable IP address is
available, the SMB client will connect to the first IP address for which a response is received. The first IP address in the list
is given a time advantage, to favor the DNS round robin sequence.
To show this in action, I created a configuration where the file server has 3 IP addresses published to DNS. I then ran a
script to copy a file to a share in the file server, then flush the DNS cache, wait a few seconds and start over. Below are the
sample script and screenshot, showing how the SMB client cycles through the 3 different network interfaces as an effect of
DNS round robin.
@ECHO OFF
:LOOP
IPCONFIG /FLUSHDNS
PING FS1 –N 1 | FIND “Reply”
NET USE F:=\\FS1\TEST >NUL
COPY C:\WINDOWS\SYSTEM32\MRT.EXE F:\
NET USE F: /DELETE >NUL
CHOICE /T 30 /C NY /D Y /M "Waiting a few seconds... Continue"
IF ERRORLEVEL 2 GOTO LOOP
:END
Using the multiple NICs of your File Server running Windows Server ... http://blogs.technet.com/b/josebda/archive/2010/09/03/using-the-multi...
6 de 12 27/08/2014 07:33
 
3.3 – SMB2 Durability
 
When the SMB2 client detects a network disconnect or failure, it will try to reconnect. If multiple network paths are
available and the original path is now down, the reconnection will use a different one. Durable handles allow the
application using SMB2 to continue to operate without seeing any errors. The SMB2 server will keep the handles for a
while, so that the client will be able to reconnect to them.
Durable handles are opportunistic in nature and offer no guarantee of reconnections. For durability to occur, the following
conditions must be met:
Clients and servers must support SMB2
Durable handles must be used (this is the default for SMB2)
Opportunistic locks (oplocks) must be used (this happens when files are opened) (this is the default for SMB2)
For Windows operating systems, SMB2 is found on Windows Vista, Windows 7 (for client OSes), Windows Server 2008 and
Windows Server 2008 R2 (for server OSes). Older versions of Windows will have only SMB1, which does not have the
concept of durability.
To showcase SMB2 durability, I used the same configuration shown in the previous screenshot and copied a large number
of files to a share. While the copy was going, I disabled Network1, the client network interface that was being used by the
SMB client. SMB2 durability kicked in and the SMB client moved to use Network3. I then disabled Network3 and the client
started using Network2. You can see in the screenshot below that, with only one of the three interfaces available, the copy
is still going.
 
4 – Clustered File Server
 
4.1 – Cluster Networks, Cluster Names and Cluster IP Addresses
 
Using the multiple NICs of your File Server running Windows Server ... http://blogs.technet.com/b/josebda/archive/2010/09/03/using-the-multi...
7 de 12 27/08/2014 07:33
In a cluster, in addition to the regular node name and IP addresses, you get additional names for the cluster itself and
every service (Cluster Group) you create. Each name can have one or more IP addresses. You can add an IP address per
public (client-facing) Cluster Network for every Name resource. This includes the Cluster name itself, as shown below:
For each File Server resource, you have a name resource,at least one IP address resource, at least one disk resource and
the file server resource itself. See the example below for the File Service FSA, which uses 3 different IP addresses
(192.168.1.101, 192.168.2.101 and 192.168.3.101).
Using the multiple NICs of your File Server running Windows Server ... http://blogs.technet.com/b/josebda/archive/2010/09/03/using-the-multi...
8 de 12 27/08/2014 07:33
Below is a screenshot of the name resource properties, showing the tab where you can add the IP addresses:
 
4.2 – How Cluster IP addresses are published to DNS
 
Note that, in the cluster shown in the screenshots, we have 5 distinct names, each of them using 3 IP addreses, since we
are using 3 distinct public Cluster Networks. You have the names of the two nodes (FS1 and FS2), the name of the cluster
Using the multiple NICs of your File Server running Windows Server ... http://blogs.technet.com/b/josebda/archive/2010/09/03/using-the-multi...
9 de 12 27/08/2014 07:33
(FS) and the names of the two Cluster File services (FSA and FSB). Here’s how this shows in DNS, after everything is
properly configured:
 
4.3 – Cluster Name and Cluster IP Address dependencies
 
When your clients are using a File Server resource with multiple routable IP addresses, you should make sure the IP
addresses are defined as OR resources, not AND resources, in your dependency definitions. This means that, even if you
lose all but one one IP address, the file service is still operational in that node. The default is AND and this will cause the
file service to fail over upon the failure of any of the IP addresses, which is typically not desired.
Using the multiple NICs of your File Server running Windows Server ... http://blogs.technet.com/b/josebda/archive/2010/09/03/using-the-multi...
10 de 12 27/08/2014 07:33
Tweet Delicious Save this on Delicious
Comments
Anonymous
3 Sep 2010 5:33 PM
Anonymous
3 Sep 2010 6:52 PM
Andrei Ungureanu
6 Sep 2010 6:31 PM
Below you can see the File Service resource with only one of the three IP addresses failed. There is an alert, but the file
service is still online and will not failover unless all IP addresses fail.
 
5. Conclusion
 
Network planning and configuration plays a major role in your File Server deployment. I hope this blog post has allowed
you to consider increasing the throughput and the availability of your File Server by enabling and configuring multiple
NICs. I encourage you to experiment with these configurations and features in a lab environment.
Ned here. Our friend and colleague Jose Barreto from the File Server development team has posted a very
New blog post covers Multi-NIC File Server configurations: When you set up a File Server, there are advantages
I'm curious about the section 3.3. I've noticed some changes in the Windows 7 DNS Round Robin behaviour that
are NOT documented and I was wondering if the failover is built in the SMBv2 code or it has something to do
with the TCP/IP stack (changing the order of the IP address in the list).
Using the multiple NICs of your File Server running Windows Server ... http://blogs.technet.com/b/josebda/archive/2010/09/03/using-the-multi...
11 de 12 27/08/2014 07:33
Jose Barreto - MSFT 11 Sep 2010 2:25 PM
Jason
18 Sep 2010 12:28 AM
Jose Barreto - MSFT 22 Sep 2010 7:47 PM
Bart
28 Sep 2010 1:54 PM
Jose Barreto - MSFT 28 Sep 2010 4:07 PM
And at 3.2 you should detail the client version as the DNS round robin behaves different based on the client,
destination address selection, subnet prioritization setting, etc
Andrei, section 3.3 is about durability, which is part of the SMB2 protocol. You can get more details about the
SMB2 changes in Windows 7 from this presentation on SMB 2.1
When using NIC teaming. Is it ok to have both adapters connected to the same switch? will there be any
switchport flooding similar to what I have seen with NLB.
Thanks.
There are multiple solutions for NIC teaming provided and supported by different hardware manufacturer (not by
Microsoft). I suggest that you ask the question about switchport flooding to the vendor providing the solution.
You could also search the web for "Link Agreggation Control Protocol" to read more about how it can be done.
I'm using the configuration as described in 2.3 on a DC with DNS installed. i'm receiving multiple NetBT Event ID
4319 Errors stating "A duplicate name has been detected on the TCP network. The IP address of the computer
that sent the message is in the data." I did not see an IP address in the data. The best I can tell is that the conflict
is with itself when I use nbtstat -n I see only the name of the DC I'm logged into. There is the one server name
for each IP addresses. If this is the case is this a problem? Thanks!
Bart. That's due to NetBIOS over TCP advertising both names on the same subnet. Not sure if you are using
NetBT/WINS for name resolution or if you're using just DNS. Many people these days disable NetBT, but some
older clients and applications might still require it. See technet.microsoft.com/.../cc782733(WS.10).aspx and
support.microsoft.com/.../299977
Using the multiple NICs of your File Server running Windows Server ... http://blogs.technet.com/b/josebda/archive/2010/09/03/using-the-multi...
12 de 12 27/08/2014 07:33