DefinIT Because if IT were easy, everyone would do it…

30Sep/09Off

Teaming NICs with ESX 3.5 and Cisco Switches in an aggregate.

Posted by Sam McGeown

Here's the setup. We have a core switch of 2 Cisco 3750s, connected together for fault tolerance as a single logical switch; we also have several ESX 3.5 hosts with 4 Gigabit Ethernet NICs installed each. The Virtual Machines will all be on VLAN 8 (reserved for internal servers) and the VMKernel will be on VLAN 107 (reserved for VMKernel traffic like VMotion).  I want to create a load balanced, fault tolerant aggregate of these four NICs over the Core Switch.

Configure ESX server's vSwitch

Configuring the vSwitch is actually pretty simple, but there are a couple of gotchas, so don't skip this bit! First thing to note is that if you are making changes to the vSwitch and the Service Console is on that vSwitch you can quite easily lock yourself out. Make sure you configure this correctly, first time! In this setup, I am adding all 4 NICs to vSwitch0, which will be the only vSwitch. I'll then use Port Groups to assign VLANs and Active/Passive configurations to the VMKernel/Service Console.

First things first then - assign the four NICs to the vSwitch. This is done in the Configuration Tab in VMware Infrastructure Client, then the Networking page. Edit the properties of your vSwitch, then select the Network Adaptor tab. Add all the NICs you wish to team in there (they may already be in there, depending on your setup). You should end up with something that looks like this (note that I've not assigned any VLAN yet):

 

Now you need to configure the NIC teaming, so edit the vSwitch Properties and under the Ports tab select the vSwitch. Click edit, and then go to the NIC teaming tab. Configure the teaming options like this:

That's the easy part over and done with! Time to move onto the Cisco!

Configuring the Cisco Core Switch

Firstly, we need to log on to the switch and enter enable mode; I'm going to assume you know how to do this - if not, you really shouldn't be attempting this setup!

Determine the switches trunk load balancing setup by using the command "show etherchannel load-balance". It should look something like this:

If the protocol is NOT src-dst-ip, then you won't be able to establish a trunk connection with the ESX server. If your protocol is not src-dst-ip, change it with the command "port-channel load-balance src-dst-ip". This now matches the "Route based on IP hash" setting you configured in ESX. Although ESX has a setting for MAC based hashing, as does the Cisco, I was unable to get it to work.

Moving on. You need to create a Port-Channel interface for the trunk (this is a virtual interface that binds the 4 GigabitEthernet interfaces together). As i've got other Port-channels in use for connections to other switches, I'm setting up port-channel 40. Move to config mode (conf t) and then enter the setup:

interface Port-channel40
 description VMTEST01 Aggregate
 switchport trunk encapsulation dot1q
 switchport trunk native vlan 8
 switchport mode trunk
 switchport nonegotiate
 spanning-tree portfast trunk
end

Description simply adds a description, "switchport trunk encapsulation dot1q" sets the encapsulation of the trunk to 802.1Q. "switchport trunk native vlan 8" means that any traffic without a VLAN tag will be automatically assigned to VLAN 8. "switchport mode trunk" obviously designates that we want a trunk, rather than access. "switchport nonegotiate" means that it will not attempt to negotiate the protocol, and be a static trunk, rather than LCAP or PGaP. "spanning-tree portfast trunk" causes a Layer 2 LAN interface configured as an access port to enter the forwarding state immediately, bypassing the listening and learning states (i.e. if the link goes down and then comes back up, it will do so quickly).

With the Port-channel configured, you now need to edit your GigabitEthernet ports and assign them to the Port-channel. For each port in the trunk, enter the following config (this example is port 8 on the master switch in my stack, hence 1/0/8):

interface GigabitEthernet1/0/8
 description VMTEST01 VMNIC1
 switchport trunk encapsulation dot1q
 switchport trunk native vlan 8
 switchport mode trunk
 switchport nonegotiate
 channel-group 40 mode on
 spanning-tree portfast trunk
end

The difference between that and the Port-channel setup? "channel-group 40 mode on" is simply assigning the port-channel in static mode.

Once all four NICs are assigned you might have to wait a few minutes for every layer of the connection to settle down before the trunk comes up. To check the status of the etherchannel you can use the command "show etherchannel 40 summary", replacing the 40 for whichever number you assigned to your port-channel.

I hope this helps navigate the minefield that I found to be setting up the NIC teaming!

21Sep/09Off

DCDIAG /TEST:DNS fails with errors regarding root hint servers

Posted by Sam McGeown

I recently resolved an ongoing DNS issue where the Active Directory Integrated DNS was loaded in both the Domain and the DomainDNSZones partition of AD - this is a separate issue and should be resolved differently. My problem when I tried to verify that the fixed DNS setup had propogated around my domain controllers, DC01 and DC02. DC01 kept failing "DCDIAG /TEST:DNS" with errors regarding the root hint servers. Googling about it was clear that a lot of people were suffering the same issue, but no article I read had correctly identified the solution.

The error looked something like this:

P:\>dcdiag /test:dns

Domain Controller Diagnosis

Performing initial setup:
   Done gathering initial info.

Doing initial required tests

   Testing server: SITE\DC01
      Starting test: Connectivity
         ......................... DC01 passed test Connectivity

Doing primary tests

   Testing server: SITE\DC01

DNS Tests are running and not hung. Please wait a few minutes...

   Running partition tests on : ForestDnsZones

   Running partition tests on : DomainDnsZones

   Running partition tests on : Schema

   Running partition tests on : Configuration

   Running partition tests on : DOMAIN

   Running enterprise tests on : DOMAIN.com
      Starting test: DNS
         Test results for domain controllers:

            DC: DC01.DOMAIN.COM
            Domain: DOMAIN.com


               TEST: Forwarders/Root hints (Forw)
                  Error: Root hints list has invalid root hint server: a.root-se
rvers.net. (198.41.0.4)
                  Error: Root hints list has invalid root hint server: b.root-se
rvers.net. (192.228.79.201)
                  Error: Root hints list has invalid root hint server: c.root-se
rvers.net. (192.33.4.12)
                  Error: Root hints list has invalid root hint server: d.root-se
rvers.net. (128.8.10.90)
                  Error: Root hints list has invalid root hint server: e.root-se
rvers.net. (192.203.230.10)
                  Error: Root hints list has invalid root hint server: f.root-se
rvers.net. (192.5.5.241)
                  Error: Root hints list has invalid root hint server: g.root-se
rvers.net. (192.112.36.4)
                  Error: Root hints list has invalid root hint server: h.root-se
rvers.net. (128.63.2.53)
                  Error: Root hints list has invalid root hint server: i.root-se
rvers.net. (192.36.148.17)
                  Error: Root hints list has invalid root hint server: j.root-se
rvers.net. (192.58.128.30)
                  Error: Root hints list has invalid root hint server: k.root-se
rvers.net. (193.0.14.129)

               TEST: Dynamic update (Dyn)
                  Warning: Dynamic update is enabled on the zone but not secure
DOMAIN.com.

         Summary of test results for DNS servers used by the above domain contro
llers:

            DNS server: 128.63.2.53 (h.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.12
7.in-addr.arpa. failed on the DNS server 128.63.2.53

            DNS server: 128.8.10.90 (d.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.12
7.in-addr.arpa. failed on the DNS server 128.8.10.90

            DNS server: 192.112.36.4 (g.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.12
7.in-addr.arpa. failed on the DNS server 192.112.36.4

            DNS server: 192.203.230.10 (e.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.12
7.in-addr.arpa. failed on the DNS server 192.203.230.10

            DNS server: 192.228.79.201 (b.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.12
7.in-addr.arpa. failed on the DNS server 192.228.79.201

            DNS server: 192.33.4.12 (c.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.12
7.in-addr.arpa. failed on the DNS server 192.33.4.12

            DNS server: 192.36.148.17 (i.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.12
7.in-addr.arpa. failed on the DNS server 192.36.148.17

            DNS server: 192.5.5.241 (f.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.12
7.in-addr.arpa. failed on the DNS server 192.5.5.241

            DNS server: 192.58.128.30 (j.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.12
7.in-addr.arpa. failed on the DNS server 192.58.128.30

            DNS server: 193.0.14.129 (k.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.12
7.in-addr.arpa. failed on the DNS server 193.0.14.129

            DNS server: 198.41.0.4 (a.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.12
7.in-addr.arpa. failed on the DNS server 198.41.0.4

         Summary of DNS test results:

                                            Auth Basc Forw Del  Dyn  RReg Ext
               ________________________________________________________________
            Domain: DOMAIN.com
               DC01                    PASS PASS FAIL PASS WARN PASS n/a

         ......................... DOMAIN.com failed test DNS


It looks pretty horrific - DNS is failing at a basic level! It turns out that the actual issue is an old version of DCDIAG.EXE. After several hours and a lot of head scratching I checked the versions of the DCDIAG.EXE (normally c:\Program Files\Support Tools\dcdiag.exe) and "Lo! And Behold!" the version was different. I downloaded the Windows Server 2003 Support Tools R2, uninstalled the old version (v5.2.3790.1800) and installed the new one (v5.2.3790.3959).

Et voila! The working test...


P:\>dcdiag /test:dns

Domain Controller Diagnosis

Performing initial setup:
   Done gathering initial info.

Doing initial required tests

   Testing server: SITE\DC01
      Starting test: Connectivity
         ......................... DC01 passed test Connectivity

Doing primary tests

   Testing server: SITE\DC01

DNS Tests are running and not hung. Please wait a few minutes...

   Running partition tests on : ForestDnsZones

   Running partition tests on : DomainDnsZones

   Running partition tests on : Schema

   Running partition tests on : Configuration

   Running partition tests on : DOMAIN

   Running enterprise tests on : DOMAIN.com
      Starting test: DNS
         Test results for domain controllers:

            DC: DC01.DOMAIN.COM
            Domain: DOMAIN.com


               TEST: Dynamic update (Dyn)
                  Warning: Dynamic update is enabled on the zone but not secure
DOMAIN.com.

         Summary of DNS test results:

                                            Auth Basc Forw Del  Dyn  RReg Ext
               ________________________________________________________________
            Domain: DOMAIN.com
               DC01                    PASS PASS PASS PASS WARN PASS n/a

         ......................... DOMAIN.com passed test DNS

10Sep/09Off

Multi-homed Domain controller logs Event ID 1030 and 1058

Posted by Sam McGeown

I recently had an issue where a hosting environment was registering a lot of Netlogon Event 1030/1058 issues, being unable to find the Group Policy objects or download them. In this example, the server DC is the domain controller for DOMAIN.LCL.

Event Type: Error
Event Source: Userenv
Event Category: None
Event ID: 1030
Date:  10/09/2009
Time:  06:24:29
User:  NT AUTHORITY\SYSTEM
Computer: DC
Description:
Windows cannot query for the list of Group Policy objects. Check the event log for possible messages previously logged by the policy engine that describes the reason for this. For more information, see Help and Support Center at
http://go.microsoft.com/fwlink/events.asp.

Event Type: Error
Event Source: Userenv
Event Category: None
Event ID: 1058
Date:  10/09/2009
Time:  06:24:29
User:  NT AUTHORITY\SYSTEM
Computer: DC
Description:
Windows cannot access the file gpt.ini for GPO CN={31B2F340-016D-11D2-945F-00C04FB984F9},CN=Policies,CN=System,DC=DOMAIN,DC=LCL. The file must be present at the location <
\\DOMAIN.LCL\sysvol\DOMAIN.LCL\Policies\{31B2F340-016D-11D2-945F-00C04FB984F9}\gpt.ini>. (Windows cannot find the network path. Verify that the network path is correct and the destination computer is not busy or turned off. If Windows still cannot find the network path, contact your network administrator. ). Group Policy processing aborted. For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

On the affected machines, when navigating to \\DOMAIN.LCL there were no shares available, however navigating to \\DC shows the NETLOGON and SYSVOL shares. Pinging DOMAIN.LCL and then the DC showed that the IP addresses were not the same as expected, DOMAIN.LCL was resolving to the backup network, whereas DC was resolving to the servers LAN IP.

I checked the DNS records for the server, which were correct. Investigating the adaptor binding settings under Control Panel > Network Connections > Advanced > Advanced Settings showed that the backup network's adaptor was first in the list. I moved the adaptor for the LAN to the top of the list and OK'd my way out. I restarted the NETLOGON service and the issue was solved.

Windows servers have never been particularly good at being multi-homed, especially domain controllers. My advice comes from some bitter experience...

  • If you have multiple network adaptors for extra bandwidth/redundancy/resiliance, then I would strongly recommend using Teamed adaptors, most of the major manufacturers' drivers and management software support it. This will eliminate any issues with multi-homing because as far as the server is concerned, it has one adaptor.
  • If you have multiple network adaptors for different network segments and you're using RRAS to route between them, I would strongly suggest not using a Domain Controller at all for this purpose. Better yet, buy a hardware router.
  • If you have multiple network adaptors for different purpose networks (e.g. a LAN, a backup network and an iSCSI network) then make sure you do the following:
    • Disable "File and Printer Sharing for Microsoft Networks" and "Client for Microsoft Networks" on all but the LAN adaptor.
    • Ensure that your LAN adaptor is the FIRST adaptor in the bindings in the advanced network settings.

 Hope that helps!

Filed under: Uncategorized No Comments