Why does it require multiple certificates for my vCenter server?

In short, each service requires a certificate because it could feasibly be on a server (or servers) of it's own - take this hypothetical design - each role is hosted on it's own VM, and there are 7 certificates required - SSO, Inventory Service, vCenter Server, Orchestrator, Web Client, Log Browser and Update Manager. If you install all these services on one server you still have to create certificates for those individual services.

Why install valid certificates at all - it "works" with the self-signed ones!

Well yes, it does work - but so does entering a blank password. That doesn't mean it's a good idea! This goes back to a fundamental understanding of what SSL does - it provides a method to secure communications between a server and a client, whether that client is another server, the same server or your vSphere Client on your laptop. In effect, the certificate says "this is who I am, this is who vouches for my identity", the Certificate Authority acts as the 3rd party that vouches for the server's identity. If you use a self-signed certificate you have a situation where "this is who I am, and I say so"!

With a self-signed certificate you can get into the bad habit of just clicking "Ignore" on a certificate warning:

I think the key line in this warning is "Secure communication with <server> cannot be guaranteed".

OK, so it's a "best practice" - but isn't the risk low? Is it worth the effort?

At the end of the day it's your call - but it's a best practice for a reason. To me security is always about mitigating risks and you have to ask yourself whether a few hours to secure your hosts is worth it. In my opinion, a good strong unique password will cost you more time over a year than installing certificates - but you wouldn't dream of using a blank password would you?!  There are tools out there like the VMware Certificate Automation Tool, and plenty of helpful walk-through's available that make it a lot less painful now.

There are also strong hints from VMware that tools like PowerCLI will be less lenient by default in future releases, which can only be a good thing.

Isn't it time you considered securing your Virtual Environment?

Enter VMware Update Manager - we already use it to patch and upgrade our hosts, but it can also apply host extensions - driver packages as well.

Identify the HBA and driver package required

First, we need to determine the exact hardware ID of the HBA and then get the latest supported driver package from the VMware Hardware Compatibility List. To do this

Log on to the host via SSH and issue an "esxcfg-scsidevs -a" to list installed SCSI devices. You'll see output similar to below - the currently loaded driver is the second column, highlighted in red.

vmhba1  qla2xxx           link-up   fc.20000024ff25da1c:21000024ff25da1c    (0:2              1:0.0) QLogic Corp ISP2532-based 8Gb Fibre Channel to PCI Express HBA

Use "vmkload_mod -s qla2xxx | grep Version" to locate the exact version of the driver loaded and you'll see output similar to this:

Version: Version 911.k1.1-26vmw, Build: 472560, Interface: 9.2 Built on: Feb  8 2012

Now to get the exact device code to look the drivers up on the HCL, use "vmkchdev -l | grep vmhba1" - this will list the Vendor,  Device, Sub-vendor and Sub-device IDs:

00:15:00.0 1077:2532 1077:0166 vmkernel vmhba1

Plug those device details along with the version of ESXi you're running into the HCL and you should get the correct device identification - the host I'm using is running ESXi 5.1 so I can see that qla2xxx version 934.5.10.0-1vmw is the correct driver. Once you've identified the correct package, download and unzip it.

Importing the package into VMware Update Manager

Unzip the package you've downloaded and locate the "offline_bundle.zip" - in my case this was a "qla2xxx-934.5.10.0-offline_bundle-1031464.zip"

Open the Update Manager administration view from your vSphere client and go to the Patch Repository tab, then click "Import Patches"

Import the offline bundle and confirm the details of the import:

Create and apply the baseline

Once the import is completed, search for the package using the Patch Repository page, and then click "Add to baseline" if you want to add it to an existing baseline, or head over to "Baselines and Groups" if you want to create a new baseline. I need to apply this to specific test hosts first, so I don't want to add to my existing baselines.

From the Baselines and Groups tab, click "Create..." and then name your baseline - it's good practice to include a decent description too. Make sure you select "Host Extension" as the baseline type.

Search for and add the patch you've just uploaded, and finish the wizard.

You can now scan, stage and remediate with the baseline as you would any other.

Hope this helps!

Sources

• Determining Network/Storage firmware and driver version in ESXi/ESX 4.x and 5.x (1027206)
• Installing async drivers on ESXi 5.x (2005205)
• VMware Compatibility Guide

Among other things it encourages you to read up on white papers, carry out lab work (Hands-on-labs), watch training and informational materials and thus rewarding you with points for you and your team. What is also great is points really do mean prizes!

I have recently become apart of a Team (Team - DefinIT) with the following well known Virtualisation bloggers and vExperts.

Barry Coombs - Virtualised Reality

Michael Poore - vSpecialist

Sam McGeown - DefinIT

This presents another great aspect to Cloud Credibility as it encourages team work with tasks and social/technical interactions.

If you haven't signed up I would strongly recommend you do so!

Fortunately , with the release of the SSL Certificate Automation Tool, VMware have gone some way to reducing the headache.

Gather all the components you need

OpenSSL installer: http://slproweb.com/products/Win32OpenSSL.html (I downloaded "Win32 OpenSSL v0.9.8y Light", even though it's a 64-bit server)

Microsoft Visual C++ 2008 Redistributable Package (x86): http://www.microsoft.com/en-us/download/details.aspx?id=29

VMware Certificate Automation Tool: https://my.vmware.com/group/vmware/get-download?downloadGroup=SSL-TOOL-10 (My VMware login required)

Install MS Visual C++ 2008 (pre-requisite for OpenSSL) Install OpenSSL - keep the defaults except for installing to c:\OpenSSL-Win32\, and install the binaries in the OpenSSL folder, not the System32.

Download the CA certificate chain from your CA

Navigate to the home page of the certificate server and click on Download a CA certificate, certificate chain or CRL.

Click the Base 64 option.

Click the Download CA Certificate chain link.

Save the certificate chain as cachain.p7b. in the c:\certs folder.

Double-click on the cachain.p7b file and navigate to C:\certs\cachain.p7b > Certificates.

Right-click on the each certificate listed and select All Actions > Export.

Click Next.

Select Base-64 encoded X.509 (.CER), and then click Next.

Save the export at C:\cert\root64.cer and click Next

For Subordinate Certificate Authorities, export as root64-1.cer, root64-2.cer etc. You need the whole chain.

My c:\cert folder now looks like this (with one subordinate CA):

Generating the SSL Certificates

This is a slightly modified version of my original script to generate SSL certs, the main difference being that we don't want to generate pfx files and we do want to create a chain.pem file. You need to create the certificate template just the same so I'll assume you can do that and come back!

Update the variables at the head of this script to reflect your environment and save it as Generate-VMCerts.ps1 in the c:\certs folder

Download it here: Generate-VMCerts.ps1

# The "short" name of the Virtual Center Server
$vC_NETBIOS = "DefinIT-VC01"
# The DNS name of the Virtual Center Server
$vC_FQDN = "DefinIT-VC01.definit.local"
# The IP address of the Virtual Center Server
$vC_IP = "192.168.1.150"
# The Certificate Authority Name
$CA_Name = "DefinIT-CA\DefinIT"
# The SHORT name of the template you defined earlier
$CA_Template = "CertificateTemplate:VirtualCenterWebServer"
# The CA certificate(s) you exported earlier, start with the issuing CA back up the chain to the root CA.
$CA_Certificates = @("c:\certs\root64-1.cer","c:\certs\root64.cer")
# Administrative email to use in the certificate
$AdminEmail = <a href="mailto:admin@definit.co.uk">admin@definit.co.uk</a>
# A working directory under which the certificates and folders will be created
$WorkingDir = "c:\certs\"
# An array of the services we will generate the certificates for
$Services = @("vCenterServer","vCenterInventoryService","vCenterSSO","VMwareUpdateManager","vCenterWebClient","vCenterLogBrowser","VMwareOrchestrator")
# The path to the openssl executable
$OpenSSLExe =  "c:\OpenSSL-Win32\bin\openssl.exe"

if (!(Test-Path $OpenSSLExe)) {throw "$OpenSSLExe required"}
New-Alias -Name OpenSSL $OpenSSLExe

$RequestTemplate = "[ req ]
default_bits = 2048
default_keyfile = rui.key
distinguished_name = req_distinguished_name
encrypt_key = no
prompt = no
string_mask = nombstr
req_extensions = v3_req

[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = digitalSignature, keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth
subjectAltName = DNS:ShortName,DNS:FQDN, DNS:IPADDRESS

[ req_distinguished_name ]
countryName = UK
stateOrProvinceName = West Sussex
localityName = Horsham
0.organizationName = DefinIT
organizationalUnitName = ORGREPLACE
commonName = FQDN
emailAddress = ADMINEMAIL

[ v3_ca ]
subjectKeyIdentifier=hash
authorityKeyIdentifier=keyid:always,issuer
basicConstraints = CA:true

[ req_attributes ]"

foreach ($CA_Certificate in $CA_Certificates) {
	$CertificateChain += Get-Content $CA_Certificate
}

if(!(Test-Path $WorkingDir)) {
	New-Item $WorkingDir -Type Directory
}
Set-Location $WorkingDir
ForEach ($Service in $Services) {
	if(!(Test-Path $Service)) {
		New-Item $Service -Type Directory
	}
	Set-Location $Service
	write-debug "$Service: Writing Template"
	$Out = ((((($RequestTemplate -replace "FQDN", $vC_FQDN) -replace "ShortName", $vC_NETBIOS) -replace "ORGREPLACE", $Service) -replace "ADMINEMAIL", $AdminEmail) -replace "IPADDRESS", $vC_IP) | Out-File "$WorkingDir\$Service\$Service.cfg" -Encoding Default -Force
	write-debug "$Service: Generating CSR"
	OpenSSL req -new -nodes -out "$WorkingDir\$Service\$Service.csr" -keyout "$WorkingDir\$Service\rui-orig.key" -config "$WorkingDir\$Service\$Service.cfg"
	write-debug "$Service: Converting Private Key"
	OpenSSL rsa -in "$WorkingDir\$Service\rui-orig.key" -out "$WorkingDir\$Service\rui.key"
	write-debug "$Service: Submitting to $CA_Name"
	certreq -submit -attrib $CA_Template -config "$CA_Name" "$Service.csr" "$WorkingDir\$Service\rui.crt"
	write-debug "$Service: Generating chain.pem"
	Get-Content rui.crt | Add-Content chain.pem
	Add-Content chain.pem $CertificateChain
	Set-Location $WorkingDir
}

Now we can run the script to generate the required certificates, open a PowerShell console, navigate to c:\certs and run Generate-VMCerts.ps1. This should run through without any errors - if you have errors make sure you fix them before moving on!

My folder now looks like this:

And under each service folder, there are the following:

Running the SSL Certificate Automation Tool

Now we're ready to follow the steps on Deploying and using the SSL Certificate Automation Tool (2041600). At this point, I took a snapshot of my virtual vCenter server! I've messed this up before - trust me, you want a fall back point! Since this vCenter is a simple install - all services on one box - it should be fairly simple.

I unzipped the Certificate Automation Tool under the c:\certs folder.

The first step is to edit the ssl-environment.bat file and carefully add the paths to the generated keys (the chain.pem and rui.key files are the ones you need). There are also a few other options like what type of SSO you are using, and some advanced optional parameters for load balancers etc. It's important to add the user ID for the SSO admin and the Virtual Center admin.

Run the ssl-environment.bat file from the command prompt to set up the variables the tool needs - there is no output from running this file.

I then ran ssl-updater.bat, selected option 1 to plan the steps. I am updating everything except update manager on this server, so I enter 1,2,3,4,5,6 and get a detailed plan of how to update the certificates. Copy and paste this into a new text file and follow it closely! Enter 9 to go back to the main menu.

Now go through each step very carefully, there's a lot of text with very similar names, check for the successful completion of each task before moving on to the next. Make sure you have handy credentials for a vCenter admin, the SSO admin and the Virtual Center database. There were 16 very similar steps on my plan, I won't bore you with a detailed explanation of all of them! The basic principle is to take the each step and run the correct process - eg for:

1. Go to the machine with Single Sign-On installed and - Update the Single Sign-On SSL certificate.

I select "2. Update Single Sign-on" and then "1. Update the Single Sign-On SSL Certificate" - you then get prompted for certificate locations, user IDs and passwords needed to perform that task.

I received an error updating the vCenter Server SSL Certificate "Cannot continue with the operation due to errors". I looked in the vc-update-ssl.log file and found the following:

Could not reload vCenter SSL Certificates [05/04/2013 - 14:00:02.31]: ""Cannot reload the vCenter Server SSL certificates. The certificate might not be unique."" [05/04/2013 - 14:00:02.32]: Deleting the new certificates and keys... [05/04/2013 - 14:00:02.32]: Restoring the original certificates and keys...         1 file(s) copied.         1 file(s) copied.         1 file(s) copied. [05/04/2013 - 14:00:02.34]: Attempting rollback... Could not reload vCenter SSL Certificates [05/04/2013 - 14:00:03.13]: ""Cannot reload the vCenter Server SSL certificates. The certificate might not be unique.""

This is a known issue and is mentioned on the KB for using the tool - "This may be caused by vpxd having multiple service IDs for the Lookup service in the vpxd.cfg file". I looked in the config file and had 3 service IDs sitting there. I found the correct service ID by using:

C:\Program Files\VMware\Infrastructure\SSOServer\ssolscli>ssolscli.cmd listServices https://:7444/lookupservice/sdk

The output contains the correct ID and after deleting the 2 spurious entries and restarting, the tool was able to install the vCenter Server SSL Certificate.

Final word...

This tool takes much of the complexity out of this job and hopefully goes a step towards helping people secure their vCenter Server installations. If there's one piece of advice for anyone doing this, it's take your time and follow every step precisely! Oh, and don't forget to delete that snapshot once you've verified everything is working OK!

 Update 14/05/2013

A great procedure for updating vShield Manager Appliance SSL is here: http://longwhiteclouds.com/2012/03/31/updating-ssl-certificate-in-vshield-manager-made-easy/

The Environment:-
3x vSphere 5.1 Hosts
2x 4port Nics 1GBe (allowing 2x iSCSi vmkernel ports per host for redundancy)
Dedicated Switching (isolated from the LAN) for iSCSi and vMotion (on seperate respective VLANs)
MSA2312i SAN G2 (with 4 Shelves)
The iSCSi Multipathing policy was set to Round Robin.
SIOC is enabled.

After a great deal of digging I resolved to contacting VMware support whom pointed me in turn to the SAN as the Host log files had the following..

<span style="color: #ff0000;">2013-02-20T14:35:38.026Z cpu8:51055)ScsiDeviceIO: 2316: Cmd(0x4124401f31c0) 0x1a, CmdSN 0x6796 from world 0 to dev "naa.600508e00000000078c4a59a76937603" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2013-02-20T14:35:38.030Z cpu8:51055)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x85 (0x4124401f31c0, 91852) to dev "naa.600508e00000000078c4a59a76937603" on path "vmhba1:C1:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE 2013-02-20T14:35:38.030Z cpu8:51055)ScsiDeviceIO: 2316: Cmd(0x4124401f31c0) 0x85, CmdSN 0xab from world 91852 to dev "naa.600508e00000000078c4a59a76937603" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
2013-02-20T14:35:38.030Z cpu8:51055)ScsiDeviceIO: 2316: Cmd(0x4124401f31c0) 0x4d, CmdSN 0xac from world 91852 to dev "naa.600508e00000000078c4a59a76937603" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
2013-02-20T14:35:38.030Z cpu8:51055)ScsiDeviceIO: 2316: Cmd(0x4124401f31c0) 0x1a, CmdSN 0xad from world 91852 to dev "naa.600508e00000000078c4a59a76937603" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.</span>

So duely armed I contacted HP support whom immediately escalated the issue internally. During this time I had a very helpful conversation with a good friend @VirtualisedReal whom pointed me in the direction of the HP MSA best practice document. I applied the subnetting configuration it suggested, which seperates the iSCSi ports A1 & B1 from A2 & B2 on seperate subnets and also configured each of the hosts 2 iSCSi vmkernel ports to point to the seperate paired iSCSi ports on the SAN.

When HP did eventually come back to me they suggested the SAN was perfectly fine, However! enough time had passed since the iSCSi port configuration change that I could already see a noticable drop in latency.

I waited another week (and since then) and I am very glad to say the latency is considerably lower with no reoccurance of the locks being lost on VM vmdk files.

The process is fairly simple, but a bit labour intensive if you're doing it manually on a large cluster.

1. Retrieve the ScsiCanonicalName for each RDM
2. Set the configuration for each RDM on each Host to "PerenniallyReserved"

Step 1 - Retrieve the ScsiCanonicalName for each RDM

This PowerCLI command lists the RDM disks attached to any VMs in a particular cluster. What we need here specifically is the SCSI canonical name (often known as the naa or eui identifier), but for the sake of sanity I suggest running it manually and examining the disks to ensure you're happy with the ones the script will set to PerenniallyReserved:

Get-VM -Location "DefinIT Lab Cluster" | Get-HardDisk -DiskType "RawPhysical","RawVirtual" | Select Parent,Name,DiskType,ScsiCanonicalName

List RDM Disks in PowerCLI

Step 2 - Set the configuration for each RDM on each Host to "PerenniallyReserved"

Once we have the list of RDM disks, we can then set the reservation via Get-EsxCli

Connect-VIServer "host01.definit.local" -Credential (Get-Credential)
$esxcli = Get-EsxCli
$esxcli.storage.core.device.setconfig($false, "naa.6005076307ffc7930000000000000109", $true)

Building Steps 1 and 2 into a script

Now it's just a simple case of constructing a script to loop through the hosts in a cluster and the RDMs on each host:

param(
	[string]	$TargetCluster,
	[PSCredential]	$RootCredentials = (Get-Credential)
)

# Create a connection object to all hosts in the Target Cluster
Get-Cluster $TargetCluster | Get-VMHost | % { Connect-VIServer $_ -Credential $RootCredentials | Out-Null }

# Find the ScsiCanonicalName for all RDM Disks attached to VMs in the Target Cluster
$RDMDisks = Get-VM -Location $TargetCluster | Get-HardDisk -DiskType "RawPhysical","RawVirtual" | Select ScsiCanonicalName

# Retrieve and EsxCli instance for each connection
foreach($esxcli in Get-EsxCli) {
	# And for each RDM Disk
	foreach($RDMDisk in $RDMDisks) {
		# Set the configuration to "PereniallyReserved".
		# setconfig method: void setconfig(boolean detached, string device, boolean perenniallyreserved)
		$esxcli.storage.core.device.setconfig($false, ($RDMDisk.ScsiCanonicalName), $true)
	}
}

# Disconnect the connection objects created for the Target Cluster
Disconnect-VIServer * -Confirm:$false | Out-Null

The script connects to each ESXi instance in the target cluster, queries all VMs on the target cluster and returns the RDM disks and then sets the PerenniallyReserved flag for each of the cluster hosts and RDM disks.

As usual, post any comments or improvements!

Edit: After my conversations with Mike, I've modified the script a little and attached as a zip file to download here: Set-RDMReservations

With 200+ blogs now up and running with content covering every aspect from PowerCLI to VDI, technical deepdives and general VMware topical blogging! there is a very strong chance you will have read an article in at least a few of them.

Last week, I had 3 vCenter Servers that appeared to be happily talking to each other in Linked Mode sharing a singe Multi-site SSO domain without any real issues. I had a single-pane-of-glass view of all 3 and I could manage them all from the one client. The reason for the 3 vCenter servers was segregation of LAN and DMZ networks: vCenter001 was in the LAN, vCenter002 sat in DMZ1 and vCenter003 sat in DMZ2.

At the weekend I rebuilt vCenter003 as a scheduled upgrade from Server 2008 to 2008R2 and from vCenter 5.1a to 5.1b, however when it came to joining the Linked Mode group, I failed each and every time with the error:

----------------- Operation "Join instance VMwareVCMSDS" failed: Action: Join Instance Action: Join Instance Action: Create replica instance Action: Create Instance Problem: Creation of instance VMwareVCMSDS failed: Active Directory Lightweight Directory Services could not create the NTDS Settings object for this Active Directory Lightweight Directory Services instance CN=NTDS Settings,CN=vCenter003$VMwareVCMSDS,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,CN={411111112-F11C-4113-9117-91111111111113} on the remote AD LDS instance vCenter001.definit.local:389. Ensure the provided network credentials have sufficient permissions. Error code: 0x800706ec The list of RPC servers available for the binding of auto handles has been exhausted.

----------------- Recovering from failed Operation "Join instance VMwareVCMSDS"

----------------- Recovery successful

----------------- Execution error.

I carefully checked the vCenter 5.1 Linked Mode prerequisites document but did not find any problem with my setup. I had in fact completed this process successfully twice before when connecting vCenter002 and the old vCenter003. After a while of troubleshooting I involved my friendly neighbourhood network support analyst who went through the ports listed by VMware with me again, and we found that the RPC Endpoint Mapper (port 135) and the high range dynamic ports were being blocked. We agreed to open 135 and a range for the dynamic ports.

Initially I limited this to a range of 1000 ports, however it quickly became clear that the rather large vCenter001 server had consumed these in seconds (each outbound connection uses a port from the dynamic pool, so each connection to a host would consume a port - multiply that by each destination port and add in the database connections and the pool needed to be considerably larger). In the end, 20,000 ports were opened for the use of dynamic ports.

At this point I was still unable to join the Linked Mode group, and the network traces were not showing any blocked network traffic. It did not appear to be a network issue!

We also tested using an IPSec tunnel between the two vCenter servers to allow all traffic without inspection at the firewall end, this is in fact how vCenter002 and vCenter001 are configured.

I engaged with VMware Tech Support and brought the technician up to speed. He asked me to go through the Troubleshooting vCenter Linked Mode document which is a great "checklist" for verifiying the requirements for Linked Mode.

In running the checks we came across lots of errors similar to:

1772 The list of RPC servers available for the binding of auto handles has been exhausted

and

DsBindWithCred to vCenter001:389 failed with status 1753 (0x6d9): There are no more endpoints available from the endpoint mapper.

His initial diagnosis was that the domain trust between the DMZ domain and the LAN domain was broken, however this was verified running through the document. The problem appeared to be related to RPC and the AD LDS (ADAM) setup. He went away to escalate the issue to the Escalation Engineers and came back to me today, hopefully he won't mind me quoting his response:

I have looked for any Kb which states officially that nesting the VC in the DMZ and using ipsec tunneling is unsupported and I cannot locate any such document , I have fully discussed this also with Escalation Engineers whom also confirm that what I am saying is correct , VMware have never certified or tested the type of deployment

VMware's guidance is to either use a LAN side vCenter server to manage the hosts through a DMZ, or to continue with the 3 vCenter servers as stand-alone servers.

At this point, troubleshooting the RPC error and the join problem become moot - I can't run an unsupported configuration in a live environment! I don't know why the new server would not join the Linked Mode group, certainly the default range for dynamic ports changed in 2008R2 and there is definitely a problem with the AD LDS installation on vCenter001.

Testing this setup through a firewall is definitely in the plan for my lab at some point and will definitely fall in the "unsupported" category, but it now leaves me with the task of pulling apart vCenter001 and 002 and ensuring they are in a stand-alone configuration.

A quick search of the web suggested removing said VM from the inventory and re-adding from the datastore, for many this fixed the issue but not for me.

Another suggested removing and reading the Host from the cluster which I did and still no joy. Finding little else to go on I elected to simply restart the host the VM/template was originally on.

Lo and behold this fixed the issue!

An error was received from the ESX host while powering on VM [VM name].
cpuid.coresPerSocket must be a number between 1 and 8

Digging around on google the error seemed to be related to over-allocating vCPUs (e.g. assigning 8 vCPUs on a VM with 4 physical CPU cores). This was a single vCPU machine on a 12 processor host, so not likely to be that! It did give me the idea that maybe the VMX had an error, so I edited the VM hardware and added an extra CPU and saved the config. I then edited it back to a single CPU and powered on the machine - it worked!

Examining the vmx showed that the coresPerSocket was set to zero which is incorrect:

cpuid.coresPerSocket = "0"

And after the change, the numvcpus setting was added and coresPerSocket updated:

cpuid.coresPerSocket = "1"
numvcpus = "1"

Fortunately, it's a simple fix and once I'd updated the template, not something that will bother me again!

VMware's official page for the Support Assistant is here - https://www.vmware.com/products/datacenter-virtualization/vcenter-support-assistant/overview.html

The OVF deploy is so simple I've just taken screenshots:

Once you've got the OVF deployed, open a console and power on the VM and set a complex root password:

Next you get presented with a network configuration script, it's just a case of entering IP addresses, DNS, hostnames etc and is all pretty straight forward, but make sure you read the instruction highlighted in red! Not doing so can cause network problems. If you need to reconfigure the network for any reason (like not following the instructions!) then log on to the console and use the "configure-network".

When that's complete, you are presented with this screen:

At this point I updated the VMware Tools using the automatic option

Once that is complete (I don't like the ! ), point your browser at the IP address configured to open the appliance, this presents you with 5 steps to install the appliance.

Step 1, registering the appliance with vCenter Server is pretty simple, just enter your server name, user ID and password (I entered it in user@domain.com format, which worked) and click register plugin.

There's only a small text message telling you anything happened at this point! My VC client prompted me to OK the certificate for the appliance, which I did. I restarted the client and verified it was installed in the plug-in manager:

Step 2 just says to "configure your network", which I would hope we did during the install!

Step 3 is installing Flash - if you're already using the Web Client then you've already got Flash installed.

Step 4 is optional, you only need to do it if you are using the Web Client - which I am! Check the file "webclient.properties" exists on your server running the Web Client - mine did - and edit the file, ensuring "scriptPlugin.enabled=true" is in the file. Then restart your Web Client.

Step 5 is to access the Support Assistant through your client!

On accessing it for the first time, it ran through a diagnostic

Then log in using your VMware support credentials and you are up and running - I was able to see my existing support requests, I haven't yet raised a new one but will update when I have.

And on the web client:

I've just seen this on the VMWare TAM blog too, great article: http://blogs.vmware.com/tam/2013/01/introducing-vmware-vcenter-support-assistant-51.html

Update 25/01/2013 14:52

It seems that viewing the Support Appliance in the web client is crippled by the certificates which were shipped with the appliance which have no Subject Name and are expired! I can access it by lowering the security zone in IE9, then chosing to ignore SSL errors and re-accessing the plugin - not ideal!

My exam experience was somewhat marred by a very poor first attempt which I narrowly failed. The exam I sat had dozens of spelling and grammatical mistakes, inaccuracies and other problems and I spent far too long commenting on those than concentrating on the questions. Fortunately I was eventually able to speak with VMware Education and they issued me with an exam voucher (they will also be releasing a new version of the exam soon, which I'm assured will resolve these problems). My second attempt was a lot better and I smashed the 300 point pass mark by 128 points, which went some way to restoring confidence in my own knowledge of the subject!

I'm now looking forward to studying for the VCAP-DCA and DCD exams with a view to completing them in 2013...

Without wishing to bore the pants off of any would be reader I shall summarize my ruminations as someone whom is still quite new to the VMware world.

The first thing that comes to mind is a a couple of recent meetings I have had with VMware. Learning that they are now very keen to engage with 'the rest of us' and by that I mean those of us working in SME's as we represent well over 50% of their business revenue. For me personally this was excellent news as we have already invested heavily into the VMware product range and plan to carry on doing so in the future. The recent release of VMware suites was a good step forward but I still feel they need to do a lot better in communicating to SME's about their vast (and ever increasing) product range as there are many gems that can often go unoticed. Our discovery of vCops earlier this year was a good example of this.

The error thrown was:

User credentials are incorrect or empty. Provide correct credentials.

After a couple of hours online with VMware support I took a guess at the problem. On the existing Single Sign On Configuration I have added the Active Directory domain DefinIT and in order to enable integrated authentication from the vSphere Client I moved it to the top of the list - this meant that System-Domain is no longer the default authentication domain. The SSO admin account (admin@System-Domain) is a part of that domain and so my guess is that the installer tries to authenticate using admin@definit.local rather than System-Domain, which of course failed.

Moving System-Domain back to the top of the list allowed me to install correctly, and once finished I could drop it back down to allow integrated authentication again.

Save the QuizMe.ps1 file into a folder and then place one or more text file in the same folder containing a comma delimited set of questions and answers. Then run QuizMe.ps1!

You can choose the quiz you take (which text file it will use).

This will work for anything you want to quiz yourself on, it's not limited to VMware configuration maxiums.

QuizMe.ps1

$QuizFiles = Get-ChildItem . | where {$_.extension -eq ".txt"}
Clear-Host
Write-Host "Choose a quiz file" -fore Yellow
foreach ($QuizFile in $QuizFiles) {
	Write-Host "$QuizFile"
}
$Quiz = Read-Host -Prompt "Type a file name"

$VMM = Import-CSV $Quiz -Header "Question","Answer"
Clear-Host
$Count = 10
while ($Count -gt 0) {
	if(($VMM).GetType().ToString() -eq "System.Object[]") {
		$Q = Get-Random $VMM
	} else {
		$Q = $VMM
	}
	$Question = $Q.Question
	$Answer = Read-Host -Prompt $Question
	Clear-Host
	if($Answer -match $Q.Answer) {
		Write-Host "Correct!" -Fore Green -NoNewLine
		$VMM = $VMM | where {$_.Question -ne $Q.Question}
	} else {
		Write-Host "Wrong - "$Q.Answer -Fore Red -NoNewLine
	}
	$temp = $VMM
	if($temp.count -eq $null) { $Count-- } else { $Count = $temp.count }
	Write-Host " $Count questions remain"
}

Download quiz files: VirtualMachineMaximums ESXi5Maximums

The script assumes that:

1. You have a working Certificate Authority
2. You are in an Active Directory domain environment
3. You have the relevant permissions to modify Certificate Templates, Request and Issue certificates.
4. You have installed OpenSSL v1.0.1c or later.

You will need to modify the configuration section to suit your environment and the $WorkingDir folder should exist before you run the script.

Creating the Certificate Template

You need to modify the standard Microsoft Web Server template to allow the encryption of user data. Since you can't edit these templates, you have to duplicate the  template and modify the settings.

Download the CA's certificate as per the instructions in the linked KB above:

• Navigate back to the home page of the certificate server and click on Download a CA certificate, certificate chain or CRL.
• Click the Base 64 option.
• Click the Download CA Certificate chain link.
• Save the certificate chain as cachain.p7b. in the c:\certs folder.
• Double-click on the cachain.p7b file and navigate to C:\certs\cachain.p7b > Certificates.
• Right-click on the certificate listed and select All Actions > Export.
• Click Next.
• Select Base-64 encoded X.509 (.CER), and then click Next.
• Save the export at C:\[Your $WorkingDir]\Root64.cer and click Next.

Running the script

The script itself is fairly simple, creating a folder structure for the individual certificates, wrestling with OpenSSL and handling the requests to the CA using CertReq. You can view the debug output by setting the $DebugPreference to 'Continue'

# The "short" name of the Virtual Center Server
$vC_NETBIOS = "DefinIT-VC01"
# The DNS name of the Virtual Center Server
$vC_FQDN = "DefinIT-VC01.definit.co.uk"
# The IP address of the Virtual Center Server
$vC_IP = "192.168.1.10"
# The Certificate Authority Name
$CA_Name = "DefinIT-CA01\DefinIT-CA"
# The name of the template you defined earlier
$CA_Template = "CertificateTemplate:VirtualCenterWebServer"
# The CA certificate you exported earlier
$CA_Certificate = "Root64.cer"
# Administrative email to use in the certificate
$AdminEmail = "admin@definit.co.uk"
# A working directory under which the certificates and folders will be created
$WorkingDir = "c:\Certificates\VMCertificates"
# An array of the services we will generate the certificates for
$Services = @("vCenterServer","vCenterInventoryService","vCenterSSO","VMwareUpdateManager","vCenterWebClient","vCenterLogBrowser","VMwareOrchestrator")# The path to the openssl executable
$OpenSSLExe =  "c:\OpenSSL-Win32\bin\openssl.exe"

if (!(Test-Path $OpenSSLExe)) {throw "$OpenSSLExe required"}
New-Alias -Name OpenSSL $OpenSSLExe

$RequestTemplate = "[ req ]
default_bits = 2048
default_keyfile = rui.key
distinguished_name = req_distinguished_name
encrypt_key = no
prompt = no
string_mask = nombstr
req_extensions = v3_req

[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = digitalSignature, keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth
subjectAltName = DNS:ShortName,DNS:FQDN, DNS:IPADDRESS

[ req_distinguished_name ]
countryName = UK
stateOrProvinceName = West Sussex
localityName = Horsham
0.organizationName = DefinIT
organizationalUnitName = ORGREPLACE
commonName = FQDN
emailAddress = ADMINEMAIL

[ v3_ca ]
subjectKeyIdentifier=hash
authorityKeyIdentifier=keyid:always,issuer
basicConstraints = CA:true

[ req_attributes ]"

if(!(Test-Path $WorkingDir)) {
	New-Item $WorkingDir -Type Directory
}
Set-Location $WorkingDir
ForEach ($Service in $Services) {
	if(!(Test-Path $Service)) {
		New-Item $Service -Type Directory
	}
	Set-Location $Service
	write-debug "$Service: Writing Template"
	$Out = ((((($RequestTemplate -replace "FQDN", $vC_FQDN) -replace "ShortName", $vC_NETBIOS) -replace "ORGREPLACE", $Service) -replace "ADMINEMAIL", $AdminEmail) -replace "IPADDRESS", $vC_IP) | Out-File "$WorkingDir\$Service\$Service.cfg" -Encoding Default -Force
	write-debug "$Service: Generating CSR"
	OpenSSL req -new -nodes -out "$WorkingDir\$Service\$Service.csr" -keyout "$WorkingDir\$Service\rui-orig.key" -config "$WorkingDir\$Service\$Service.cfg"
	write-debug "$Service: Converting Private Key"
	OpenSSL rsa -in "$WorkingDir\$Service\rui-orig.key" -out "$WorkingDir\$Service\rui.key"
	write-debug "$Service: Submitting to $CA_Name"
	certreq -submit -attrib $CA_Template -config "$CA_Name" "$Service.csr" "$WorkingDir\$Service\rui.crt"
	write-debug "$Service: Generating PFX"
	OpenSSL pkcs12 -export -in "$WorkingDir\$Service\rui.crt" -inkey "$WorkingDir\$Service\rui.key" -certfile "$WorkingDir\$CA_Certificate" -name "rui" -passout pass:testpassword -out "$WorkingDir\$Service\rui.pfx"
	Set-Location $WorkingDir
}

Multisite deployments are  where a local replica is maintained at remote sites of the primary vCenter Single Sign-On instance. vCenter Servers are reconfigured to use the local vCenter Single Sign-On service and reduce authentication requests across the WAN. Multisite deployments do drop the support of single pane of glass views unless Linked Mode is utilized and multisite deployments are actually required to maintain Linked Mode configurations where roles, permissions and licenses are replicated between linked vCenter servers. Linked mode will re-enable single pane of glass views across multisite instances.

Using two test vCenters, Definit-VC01 and Definit-VC02, I ran through the installation.

Installing the first vCenter SSO

This is important! Don't install using the Simple Installation, even if you are just using a single server. The reason for this is that you can't install SSO in Multi-Site Mode if you do. If you have already installed the vCenter Server (as I had) then you will need to uninstall all the components, preserving the database and re-install using this guide. Uninstall components in this order: Web Client, vCenter Server, Inventory Service, Singe Sign On. I also will use the same service account user for both installations.

Install Single Sign On in Multi-site mode and select to install as a new primary node.

Create an SSO administrator password and then either choose to install a new SQL Express instance, or use a supported one. I have a separate SQL server for this VC, so I will create an RSA database manually.

Modify the two scripts to reflect your installation path and passwords you want for the RSA DB.

Once you've edited and run the two scripts check that they exist and then proceed to configure the database connection.

Enter the FQDN and service details - I installed here with the Network Service account and later modified it to use a service account.

I accepted the default location, and port.

And finished the Wizard.

Stringing it all together - Linked Mode

In my previous post, Configuring vCenter Linked Mode, I ran through the process of installing Linked Mode using a single instance of SSO across a fast link, so I'm not going to cover the actual installation of Linked Mode here. The process is exactly the same and should be run using the same account that is running services on both vCenters.

The LUNs had been unpresented at the hardware level by the storage team, but had not been unmounted or removed from vCenter. This is *not* the way to remove storage - let me re-iterate: remove storage properly. Unfortunately in this case the storage was removed badly - doing this can lead to a condition called "All Paths Down" or APD which is best explained by Cormac Hogan (@vmwarestorage) in the article Handling the All Paths Down (APD) condition.

When adding the hosts back into the cluster, I received the following message:

Reconnect host HostFQDN - Datastore 'DatastoreName' conflicts with an existing datastore in the datacenter that has the same URL (ds:///vmfs/volumes/4f00008-4200009c-0000-5000000ba397/), but is backed by different physical storage.

Looking at the datastore name I realised that the 3 datastores that were causing this issue had been part of a Storage Cluster, so I attempted to remove the storage - but faced this error:

Call "HostDatastoreSystem.RemoveDatastore" for object "datastoreSystem-119955" on vCenter Server "DefinIT-VC01" failed. Cannot remove datastore 'DefinIT-006' because storage I/O Control is either enabled or running in statistics collection mode on it. Correct the problem and retry the operation.

I also attempted to connect directly to the host using the Client and remove the storage - the same error occured. Connecting to the host in Tech Support Mode via SSH and removing with PowerCLI produced similar messages.

However, I already had the clue I needed to fix the problem - the error message states "storage I/O Control is either enabled or running in statistics collection mode". Each LUN was marked by SIOC because it was part of a Storage Cluster that leveraged StorageDRS with the I/O metric enabled. I verified this using PowerCLI - "Get-DataStore 'DefinIT-006' | fl" - you can see that StorageIOCOntrolEnabled is True and State is Unavailable.

The only way to remove them from this would be to remove them from the Storage Cluster. This would work if the hosts were connected to vCenter, but they were not - and I couldn't get them into vCenter BECAUSE of this condition! Catch22.

Googleing "manually disable sioc esxi" led me to a virtuallyGhetto article about enabling SIOC without vCenter in which the solution was found!

Manually Disable SIOC on a disconnected LUN and Remove from host

I connected to a host via SSH and began...

First, I needed to find the Device Name of the missing storage using the following command:

esxcli storage vmfs extent list | grep 'Definit-006'

Armed with the naa (or eui) name for the device I could then get the iormState value for the device:

vsish -e get /storage/scsifw/devices/eui.001738004e4c0068/iormState

This returned a value of 1597 - SIOC is enabled! According to the virtuallyGhetto article, a value of 1956 will disable SIOC, so I used the set command to write the value:

vsish -e set /storage/scsifw/devices/eui.001738004e4c0067/iormState 1596

This completed, I could now move on and attempt to remove the LUN (note that it takes a few seconds to update the setting, don't attempt to remove the device too quickly!):

esxcli storage core device set -d eui.001738004e4c0067 --state=off

After this, you can either manually rescan all HBAs on the host or I found that after a few minutes they disappeared from the host. Reconnecting the host was a success! I could see in the vCenter Client the progress of my work:

This procedure was required for each LUN on each host that was disconnected and in this state - fortunately only 4 hosts required this work.

Naturally I began the process of cross referencing backup routines and any heavy I/O routines that may have been running at the time the warning messages were generated. My conclusion was that even under average load these alerts were being generated, which was far from ideal even if we had not noticed any performance problems with any of the busy VMs.

After consulting the web/reference material and a few very knowledgable friends it was clear the first port of call was the Host Datastore Multipath policy. Upon quick inspection, all of the offending Datastores were configured with the Path Selection "Most Recently Used (vmware)". I had the option to set the Path Selection to "Round Robin (vmware)" but before doing so I double checked our MSA2312i SAN could support such a policy, which in this case it did.

For the purposes of this, let's call them DefinIT-VC01.definit.co.uk and DefinIT-VC02.definit.test.

Linked Mode Pre-requisites

As well as the normal vCenter Server pre-requisites, there are certain criteria you have to fulfil to be able to use linked mode too. These are taken from the docs here: http://pubs.vmware.com/vsphere-51/index.jsp?topic=%2Fcom.vmware.vsphere.install.doc%2FGUID-7C9A1E23-7FCD-4295-9CB1-C932F2423C63.html

Linked Mode groups that contain both vCenter Server 5.x and versions of vCenter Server earlier than 5.0 are not supported. The vSphere Client does not function correctly with vCenter Servers in groups that have both version 5.x and pre-5.0 versions of vCenter Server. Do not join a version 5.x vCenter Server to pre-5.0 versions of vCenter Server, or pre-5.0 version of vCenter Server to a version 5.x vCenter Server. Upgrade any vCenter Server instance to version 5.0 or later before joining it to a version 5.x vCenter Server.

Easy one to start with, both my vCenter Servers are 5.1!

Make sure that all vCenter Servers in a Linked Mode group are registered to the same vCenter Single Sign On server.

Not so easy - I need to register DefinIT-VC02 with DefinIT-VC01's Single Sign On service. This is possible using Repointing and reregistering VMware vCenter Server 5.1.x and components, however if your install path is not the default the scripts do not work. There are references all over the place to "c:\Program Files" hard coded - even editing them as best I could I couldn't get it to work. In the end I removed everything from DefinIT-VC02 and reinstalled each component, skipping the SSO and pointing the Inventory Service, vCenter Server and WebClient to DefinIT-VC01.

To join a Linked Mode group the vCenter Server must be in evaluation mode or licensed as a Standard edition. vCenter Server Foundation and vCenter Server Essentials editions do not support Linked Mode.

Both VCs are installed with a vCenter Server Standard License.

DNS must be operational for Linked Mode replication to work.

Both VCs can resolve their own and each other's DNS names.

The vCenter Server instances in a Linked Mode group can be in different domains if the domains have a two-way trust relationship. Each domain must trust the other domains on which vCenter Server instances are installed.

The two domains do trust each other.

When adding a vCenter Server instance to a Linked Mode group, the installer must be run by a domain user who is an administrator on both the machine where vCenter Server is installed and the target machine of the Linked Mode group.

Both vCenter Servers are run using the same service account, and the Linked Mode Configuration will be run from this context.

All vCenter Server instances must have network time synchronization. The vCenter Server installer validates that the machine clocks are not more than five minutes apart. See Synchronizing Clocks on the vSphere Network.

Finally, check you're in time sync - 5 minutes is a big difference though, more tolerant than most applications!

Configuring Linked Mode

Running the Linked Mode Configuration tool was pretty straightforward, though it did get confused in the middle reporting errors that didn't exist. The steps are self-explanatory:

There is a wizard for converting custom attributes to tags, but it can get a bit confusing and is pretty poor - let me explain. We use four custom attributes in my current environment: CreatedBy, CreatedOn, Owner and ServiceType. CreatedBy contains the user ID of the person who created the VM, CreatedOn is the timestamp of when the VM was created, Owner is the Business Unit who own the server and ServiceType is the type of service - e.g. Active Directory, or SQL.

CreatedBy is a fairly limited set of user IDs that don't change often, which I assumed lends itself quite well to the use of tags. The problem arises when you attempt to convert the attributes to tags. I ran the Wizard, filtered and selected all the CreatedBy Custom Attributes and moved on to the Create Tags page. This lists this Custom Attribute for each machine, and it's value, so clicked Next. I did this on two occasions and had two results - the first time the Wizard tried to validate the data and threw an error requiring each Tag to be unique, the second time it created two unique Tags and threw an error when it tried to create a new Tag that wasn't unique.

The problem is that the Wizard does not merge the custom attributes together when creating the tags, it tries to create a unique tag per custom attribute assigned. This is a seriously half-baked implementation! "Owner" and "Service Type" face the same problem as "CreatedBy" - there is no logical merging of attribute values to create Tags. I can create the Tags by carefully selecting one of each value, but I then need to manually assign the new Tags to each VM which would take an untenable amount of manual labour.

It also becomes a problem when you think of the CreatedOn field, which contains as many unique values as we have Virtual Machines. This isn't manageable as we'd need to create a new Tag every time there was a new timestamp. We decided to abandon this attribute as it's of limited value and there are other ways to determine when the VM was created (e.g. server commissioning forms).

The solution? At present there is no way to programattically add or remove, or assign tags and categories. This was confirmed to me by Alan Renouf (@alanrenouf) on twitter, and Luc Dekens (@LucD) on the VMware Communities forums. LucD pointed to an article on VirtuallyGhetto regarding the Managed Object Browser (MOB) which allows for the creation of tags, but this doesn't help me much.

@sammcgeown @lucd22 no unfortunately not, it's a private API at the moment

24/09/2012 3:53 pm via Twitter for iPhoneReplyRetweetFavorite

@alanrenouf

Alan Renouf

All in all, I love the new feature of Tags, it shows that VMware have thought about even this small portion of functionality (e.g. the cardinality, see below) - I just hope the wizard gets improved (or at least the PowerCLI team can access the APIs!) I also love the new web client, and despite some "we don't like change" rumblings from people around me, they are all getting used to it and enjoying it.

A small explanation of the Cardinality of tags.

There are some tags which you only need one value assigned to an object, for example if you used "Priority" as a Category and "High", "Medium" and "Low" as the available tags, you don't want to be able to tag a VM with both "High" and "Low". For this I would set the Category Cardinality for "Priority" to "1 tag per object". Conversely, there are some Categories which it would be great to be able to use multiple, for example my "ServiceType" category. I may have a Domain Controller which is also a DNS, SQL and DHCP server - I can add all those tags if I set the Cardinality to "Many tags per object".

Several months ago we relocated and it was then necessary to setup a Site to Site VPN tunnel with another network. (In this instance the other network was not directly managed by us)

Upon the creation of the tunnel and after successful traffic tests all looked well. However after several hours or less in some cases traffic stopped flowing yet both firewalls reported the tunnel as “up”. We reviewed the first and second phase settings and tweaked the Sonicwall VPN settings to hopefully remedy.

Options on the Sonicwall such as “Enable IKE Dead Peer Detection” & “Enable Keep Alive” were enabled and disabled to try and find a fix for the VPN traffic flow problem.

What was interesting during the troubleshooting process, we found that if we manually restarted the VPN tunnel it would resume with no issue, but obviously this was hardly a practical fix for our issues.

Liaising with the other site we also experimented with Phase 1 and Phase 2 Life Time settings with no success.

It was then we had a small eureka moment, we decided to check the time servers each firewall referenced. It transpired the Time Server being referenced by the Cisco Firewall was out of sync (it was an internally hosted NTS)

After the offending NTS had been re-sync’d we decided to completely recreate the VPN tunnel double checking the settings as we went along. The VPN Tunnel came up with no issues and has been stable ever since.

I would add if we encounter a problem like this again I would simply point both Firewalls to the same NTS but as one of the firewalls in this case was managed by a third party this was not an option.

vSphere HA agent for host [Host's Name] has an error in [Cluster's Name] in [Datacenter's Name]: vSphere HA agent cannot be correctly installed or configured

After a few basic checks, migrating the host in and out of the cluster and rebooting, I headed off to google and began troubleshooting.

Cannot install the vSphere HA (FDM) agent on an ESXi host - this article suggests that the host is in lockdown mode. This is unlikely since we don't use lockdown mode, but I checked anyway:

Get-vmhost esxi001.definit.co.uk | select Name,@{N="LockDown";E={$_.Extensiondata.Config.adminDisabled}} | ft -auto Name,LockDown

This returned false - no lockdown.

To exit lockdown mode, you can use:

(get-vmhost esx001.definit.co.uk | get-view).ExitLockdownMode()

I spent a good amount of time going through the list on Troubleshooting VMware High Availability (HA) in vSphere which isn't entirely ESXi relevant but has some good pointers nonetheless.

I finally got to Reconfiguring HA (FDM) on a cluster fails with the error: Operation timed out, with the following gem of info:

 This issue occurs if the vSphere High Availability Agent service on the ESXi host is stopped.    

*Facepalm* - I checked the services and set the service to start and stop automatically. HA is now happily configured.

No matter how much you know, you gotta check the basics!

Power On virtual machine <VM name> Cannot initialize property ' vami.DNS0.vSphere_Man- agement_Assistant_(vMA)' , since network '<network name>' has no associated IP pool configuration.

Edit the vMA virtual machine's properties and go to Options, vApp Options and select disable. Acknowledge the warning and click OK to close the VM properties.

The vMA booted fine after that - the solution comes from this vmware communities post.

Simply because of the size and age of the environment, some of our production clusters have now reached the limit (including local paths) - you see this message in the logs

[2012-08-20 01:48:52.256 77C3DB90 info 'ha-eventmgr'] Event 2003 : The maximum number of supported paths of 1024 has been reached. Path vmhba3:C0:T4:L0 could not be added.

In this state the hosts drop SAN LUNs because they can't drop the locally attached storage, even though we are not using it. You can see how many paths are being used on a specific host by selecting the host, going to Configuration > Storage Adapters and selecting the local storage adaptor:

In fact you can see that this IBM blade is registering two local paths, one for the storage and one for the ServeRAID and one for the disk enclosure. Removing some LUNs reduced the number of paths and, now that it's fixed, if I look at the Fibre Channel HBAs you can see that each one has 126 LUNs and 4 targets making 504 paths, there are two Fibre HBAs and 2 local paths making a total of 1010 paths (504*2 + 2 = 1010).

Before we removed some LUNs, one adaptor had 512 LUNs and the other 510 - if you add the local adaptor paths that makes you 1024 paths (512 + 510 + 2 = 1024). This meant that the LUN in the error message had been dropped to satisfy the limit.

Of course, you want to avoid being this close to the limit: but if like me there are compelling requirements to be this close, make sure you factor in the local storage!

That's fine - there is an option to remove those modules when you remediate the host.

Part 1 - Upgrading doesn't go entirely to plan

I went ahead and remediated my first host, which was duly placed in maintenance mode and then bombed out at 7% complete with the error:

Software or system configuration of host <hostname> is incompatible. Check scan results for details.

Searching for this error led me to this Communities post  which in turn pointed to a VMware KB which recommends disabling the Ethernet over USB interface on the blade configuration: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2006133

The Ethernet over USB interface is only used if you have the IBM management agents installed - this cluster did not.

Once I'd change this the Upgrade worked on 1 out of 4 blades -  the remaining 3 threw this error (at the annoyingly late stage of 92%!):

Cannot execute upgrade script on host.

A further search took me to Upgrading from ESXi 4.0 Update 1 or Update 2 to ESXi 5.0 using vCenter Update Manager fails with the error: Cannot run upgrade script on host. I didn't have the errors in the vua.log, but following the procedure did resolve the issue on a further 2 of the hosts.

The resolution is to SSH to the host in question and browse to the /bootbank folder. List the contents and you see a state.xxxxxxx folder - move inside the folder and you see a local.tgz file. The KB tells you to move that file to the parent folder and try again.

The final hurdle to upgrading was to remove the space in the name of the management network on the remaining host. Upgrading from ESXi 4.x to ESXi 5.0 using Update Manager fails with the error: integrity.fault.HostUpgradeRunScriptFailure points to the port group name for the Management Network needing to be less than 20 characters - which the default ("Management Network")  is. Since our management network is left to default, I moved on - but later read a forum post mentioning the need to remove spaces.

Finally the install process was working - however there were other issues once I'd upgraded!

Part 2 - MSCS RDM LUNs and ESXi 5 cause an incredibly long boot time

The hosts took an exceedingly long time (read 1-2 hours) to progress past "vmw_satp_alua loaded successfully" - Alt-F12 showed that the boot was going forward but I'm talking 3 hours in total to boot!  Subsequent reboots did not improve the situation, and this wasn't an issue with the ESXi 4 installation.  We have RDMs attached to these hosts for MSCS, and this led me to this article: ESX/ESXi hosts hosting passive MSCS nodes with RDM LUNs may take a long time to boot. (If you use iSCSI this VMware KB is more relevant).

All four of hosts have the RDM luns presented and although only two of the hosts actually run the Microsoft Cluster Services, the slow boot affected all four. The problem is that when the host boots it tries to claim all LUNs that it can see - including the RDMs which are reserved for the Cluster. As the doc puts it better than me:

ESXi 5.0 uses a different technique to determine if Raw Device Mapped (RDM) LUNs are used for MSCS cluster devices, by introducing a configuration flag to mark each device as "perennially reserved" that is participating in an MSCS cluster. During a boot of an ESXi system the storage mid-layer attempts to discover all devices presented to an ESXi system during device claiming phase. However, MSCS LUNs that have a permanent SCSI reservation cause the boot process to elongate as the ESX cannot interrogate the LUN due to the persistent SCSI reservation placed on a device by an active MSCS Node hosted on another ESXi host.

Following the procedure in the VMware KB, I found the naa.xxx ID for each the 14 RDM LUNs (and had to manually type them out, grrr) and then enabled SSH on the newly upgraded hosts to send the following command:

esxcli storage core device setconfig -d naa.6005076307ffc7930000000000000103 --perennially-reserved=true

and verified it with the command

esxcli storage core device list -d naa.6005076307ffc7930000000000000103

Rebooting then took about 10 minutes, most of which was the UEFI. Unfortunately this process needed to be run on each host, and each host had to boot with the RDM LUNs connected to then be able to change their reservation. Unfortunately, with each host taking several hours to boot through the 14 RDMs, stepping through to upgrde the cluster hosts took an awful long time! I'm not looking forward to doing the "big" clusters with 20, 30 and 50 hosts.

Sam

Fortunately, IPSec is allowed between each DMZ and the management DMZ so the plan is to configure IPSec between a new Enterprise Management Server in the Management DMZ (we''ll call it EMS01) and each of the three TMG servers.

Configuring the IPSec connection

There's nothing fancy about this, just a plain ol' IPSec setup:

Create a new IPSec Policy using the wizard

The policy does not specify a tunnel (it's host-to-host) and we can keep it set to all network connections

Add an IP Filter list and name it accordingly - click Add to start the IP Filter Wizard and create an entry description from EMS01 to TMG01 - ensure the Mirrored option is checked.

Add the IP address of EMS01 as the source, and TMG01 as the destination

We want to encrypt any traffic between these servers, and finish the wizard. Repeat this process (using the IP Filter Wizard) to add entries for all of the servers you wish to secure communications between. I created three rules, "EMS01 to TMG01", "EMS01 to TMG02" and "EMS01 to TMG03". Ensure the filter just created is selected and click Next.

Create a filter using the Filter Action wizard, named appropriately and set to negotiate security

Deny unsecured communication and require the Integrity and encryption option.

As all servers are domain members we can use Kerberos authentication. Finish the wizards

. 

With the policy now created but unassigned, right click "IP Security Policies on Local Computer" and select "All Tasks > Export Policies...". Copy the created .ipsec file to the other servers in the array and import the file.

Add the new TMG EMS server to the Array Servers group within the existing TMG array and create a new rule allowing IPSec (add "IKE Client" and "IPSec ESP") from and to "Array Servers" and "localhost" - this is critical as without it, we will not be able to communicate between these servers.

Finally, we are ready to "Assign" the IPSec policies on all TMG servers - I assigned it on EMS01 and TMG01 first and fired a ping to check the IPSec communication was working.

Troubleshooting note: you can add "IP Security Monitor" snap-in to an MMC console to monitor the IPSec communication.

 Installing the TMG Enterprise Management Server

Run the Preparation Tool, selecting the Enterprise Management Server option and ensure all the pre-requisites are installed.

Accept the EULA and enter your license and company details. Set the installation path and then select the option to Create a new enterprise configuration.

Enter a name and description for the Enterprise

Since all my TMG servers are in a single domain, I will chose the Single Domain Deployment.

Hit install and watch the blue bar until completion

Firing up the console you can see the Enterprise configuration and Arrays.

Log on to the TMG stand-alone array and export the configuration to file.

Run through the Export Wizard, ensuring that you export the confidential information and password protect it

Copy the exported config file over to EMS01, log on to the server and open the TMG console.

Importing the TMG Array Configuration to the Enterprise Management Server

Right click Arrays and select "New Array..."  and then match the array name exactly to the existing array

Match the Array DNS name exactly, and select the Default Policy

Ensure that all 3 policy types are checked and finish the wizard, but don't apply the configuration yet! Right click on the array and select "Import (Restore)..."

Select the file we created before and select the option to "Overwrite (restore)"

Select "Import server-specific information" and "Import user permission settings" and enter the password

If you get the following error, ensure the EMS is at the same patch level as the array servers

Error 0xc0040399 - The XML file cannot be imported to this version of Forefront TMG. Install the latest version of Forefront TMG updates and then try importing the XML file again.

Once the import is completed, save the array configuration. You will be prompted to restart the services - select Save the changes and restart the services.

At this point the stand-alone array will not be affected - it's still pointing at it's own configuration storage on the array servers. If you look at the System tab, the connection to the servers will either have an hour glass, or a red cross - this is to be expected, they are not configured to talk to the EMS yet. The Config Management field will be set to "EMS-Managed Array"

Disjoining Servers from the Stand-alone Array and joining the EMS Managed Array

Disjoining the servers will erase the config on the servers, so it needs to be done carefully and out of business hours. Disjoining the non-CSS server first will leave the current array manager in a working state while the other is disjoined and joined to the EMS. If you are using NLB, both servers will remain as part of the NLB while this happens - this can mean that traffic will be directed to the disjoined array but denied by the default rule.

Log directly onto the TMG server that is not the array manager, and open the TMG Console. Select the top node and in the actions pane, click the "Disjoin Server from Array" link.

 Now we have one of the array disjoined and ready to join the EMS managed array. Since the TMG config is essentially blank, create a rule to allow IPSEC again between the EMS and this server. Now run the "Join Array" wizard

Enter the FQDN of the EMS and select the array you have just configured

And that's it - joined! Repeat the process for the other array member

Check you have the servers correctly in the array, and the configuration is syncing correctly

Since this post is getting long, I'll split adding the 3rd server for another day - I'll update here when it's done.

Updated for this version

• Formatting changed to make it more readable and more compatible
• Added "Report generated on <server>" to the top of the report
• Management Server states reported as one section
• Default MP check moved to beneath the Management servers
• Agents in pending states moved to be with the Agent health states
• Clarified "Unresponsive Agents" and "Agents reporting errors"
• Management server alerts streamlined
• Added top 10 alerts for the last 7 days, and added top alerters for each

I'm planning to wrap in some SQL database size checks and some of the other recommendations later - I'll post again here when that's ready

$Head = "<style>"
$Head +="BODY{background-color:#FFF;font-family:Verdana,sans-serif; font-size: 11px;}"
$Head +="TABLE{border-width: 0px;border-collapse: collapse; width: 100%;}"
$Head +="TH{border-width: 0px;background-color:#F5F5F5;color:Navy;padding: 5px; font-weight: bold;text-align:left;}"
$Head +="TD.Gray{ border-width: 0px; background-color: #D3D3D3; color:Navy; padding: 5px; font-weight: bold; text-align:left;}"
$Head +="TD.Blank{ border-width: 0px;}"
$Head +="TD{ border-width: 0px; color:Red;}"
$Head +="H1{color:Navy; font-size: 12px; padding: 5px;}"
$Head +="H3{color:Navy; font-size: 11px;}"
$Head +="P{font-size: 11px;}"
$Head +="P.OK{color:Green; font-size: 11px;}"
$Head +="P.Error{color:Red; font-size: 11px; font-weight: Bold;}"
$Head +="</style>"
$ReportOutput += "<H1>SCOM Daily Healthcheck Report</H1>"
$ReportOutput += "<p>Report generated on "+(gwmi WIN32_ComputerSystem).Name+"</p>"

write-host "Getting Management Health Server States" -ForegroundColor Yellow
$ReportOutput += "<TABLE><TR><TD class=Gray><H1>Management Servers</H1></TD></TR><TR><TD class=Blank>"
$Count = Get-ManagementServer | where {$_.HealthState -ne "Success"} | Measure-Object
if($Count.Count -gt 0) {
	$ReportOutput += Get-ManagementServer | where {$_.HealthState -ne "Success"} | select Name,HealthState,IsRootManagementServer,IsGateway | ConvertTo-HTML -fragment
} else {
	$ReportOutput += "<p class=OK>All management servers are in healthy state.</p>"
}
write-host "Getting RMS Maintenance Mode" -ForegroundColor Yellow
$RMS = Get-ManagementServer | where {$_.IsRootManagementServer -eq $True}
$criteria = new-object Microsoft.EnterpriseManagement.Monitoring.MonitoringObjectGenericCriteria("InMaintenanceMode=1")
$objectsInMM = (Get-ManagementGroupConnection).ManagementGroup.GetPartialMonitoringObjects($criteria)
$is = "<p class=OK>"+ $RMS.Name +" is not in maintenance mode</p>"
foreach ($MM in $objectsInMM){
	if($MM.Displayname -eq $RMS.Name){
		$is = "<p class=Error>"+ $RMS.Name +" is in maintenance mode</p>"
	}
}
$ReportOutput += "</TD></TR><TR><TD class=Blank>"+$is+"</TD></TR><TR><TD class=Blank>"

$ReportOutput += "</TD></TR><TR><TD class=Blank><p> </p></TD></TR><TR><TD class=Blank>"

write-host "Getting Overrides in Default Management Pack" -ForegroundColor Yellow
$ReportOutput += "</TD></TR><TR><TD class=Gray><H1>Overrides in Default Management Pack</H1></TD></TR><TR><TD class=Blank>"
$OverrideCount = Get-ManagementPack | where {$_.DisplayName -match "Default Management Pack"} | get-override | measure-object
if($OverrideCount.Count -gt 2){
	$ReportOutput += "<p class=Error>There are unexpected overrides in the Default Management Pack</p>"
	foreach ($monitor in Get-ManagementPack | where {$_.DisplayName -match "Default Management Pack"} | get-override | where {$_.monitor}) {
		$ReportOutput += get-monitor | where {$_.Id -eq $monitor.monitor.id} | select-object DisplayName,Description | ConvertTo-HTML -fragment
		$ReportOutput += "<br />"
	}
	foreach ($rule in Get-ManagementPack | where {$_.DisplayName -match "Default Management Pack"} | get-override | where {$_.rule}) {
		$ReportOutput += get-rule | where {$_.Id -eq $rule.rule.id} | select-object DisplayName,Description | ConvertTo-HTML -fragment
		$ReportOutput += "<br />"
	}
} else {
	$ReportOutput += "<p class=OK>There are no unexpected overrides in the Default Management Pack</p>"
}

$ReportOutput += "</TD></TR><TR><TD class=Blank><p> </p></TD></TR><TR><TD class=Blank>"

write-host "Getting Agents in Pending State" -ForegroundColor Yellow
$ReportOutput += "</TD></TR><TR><TD class=Gray><H1>Agents in Pending State</H1></TD></TR><TR><TD class=Blank>"
#$ReportOutput += Get-AgentPendingAction | sort AgentPendingActionType | select AgentName,ManagementServerName,AgentPendingActionType | ConvertTo-HTML -fragment
$Pending = Get-AgentPendingAction
if($Pending.Count -gt 0) {
	$ReportOutput += $Pending | sort AgentPendingActionType | select AgentName,ManagementServerName,AgentPendingActionType | ConvertTo-HTML -fragment
} else {
	$ReportOutput += "<p class=OK>No pending agents</p>"
}

$ReportOutput += "</TD></TR><TR><TD class=Blank><p> </p></TD></TR><TR><TD class=Blank>"

$ReportOutput += "</TD></TR><TR><TD class=Gray><H1>Unresponsive Agents</H1></TD></TR><TR><TD class=Blank>"
$AgentMonitoringClass = get-monitoringclass -name "Microsoft.SystemCenter.Agent"
$ReportOutput += Get-MonitoringObject -monitoringclass:$AgentMonitoringClass | where {$_.IsAvailable -eq $false} | select DisplayName | ConvertTo-HTML -fragment

$ReportOutput += "</TD></TR><TR><TD class=Blank><p> </p></TD></TR><TR><TD class=Blank>"

write-host "Getting Agent Health Status" -ForegroundColor Yellow
$ReportOutput += "</TD></TR><TR><TD class=Gray><H1>Agents reporting errors</H1></TD></TR><TR><TD class=Blank>"
$ReportOutput += Get-Agent | where {$_.HealthState -ne "Success"} | select Name,HealthState | ConvertTo-HTML -fragment

$ReportOutput += "</TD></TR><TR><TD class=Blank><p> </p></TD></TR><TR><TD class=Blank>"

write-host "Getting Management Server Alerts" -ForegroundColor Yellow
$ReportOutput += "</TD></TR><TR><TD class=Gray><H1>Management Server Alerts</H1></TD></TR><TR><TD class=Blank>"
$ManagementServers = Get-ManagementServer
$ReportOutput += "<TABLE>"
foreach ($ManagementServer in $ManagementServers){
	$ReportOutput += "<TR><TD class=Blank><H3>" + $ManagementServer.ComputerName + "</H3></TD><TD>"
	$MSAlerts = get-alert -Criteria ("NetbiosComputerName = '" + $ManagementServer.ComputerName + "'") | where {$_.ResolutionState -ne '255' -and $_.MonitoringObjectFullName -Match 'Microsoft.SystemCenter'}
	if(($MSAlerts).Count -gt 0) {
		$ReportOutput += $MSAlerts  | select TimeRaised,Name,Description,Severity | ConvertTo-HTML -fragment
	} else {
		$ReportOutput += "<p class=OK>No Alerts</p>"
	}
	$ReportOutput += "</TD></TR>"
}
$ReportOutput += "</TABLE>"

$ReportOutput += "</TD></TR><TR><TD class=Blank><p> </p></TD></TR><TR><TD class=Blank>"

write-host "Getting Top 10 Alerts in the last 7 days" -ForegroundColor Yellow
$ReportOutput += "</TD></TR><TR><TD class=Gray><H1>Top 10 Alerts (last 7 days)</H1></TD></TR><TR><TD>"
$ReportOutput += "<TABLE>"
$topten = get-alert | where {$_.TimeRaised -gt ((get-date).adddays(-7))} | group-Object Name | sort-object Count -desc | select -first 10 Name, Count
foreach ($toptenalert in $topten) {
	$ReportOutput += "<TD class=Gray>"+$toptenalert.Name+" (Total: "+$toptenalert.Count+")</TD><TR><TD>"
	$ReportOutput += get-alert | where {$_.Name -eq $toptenalert.Name -and $_.TimeRaised -gt ((get-date).adddays(-7))} | group-Object PrincipalName | sort-object Count -desc | select -first 10 Name, Count | ConvertTo-HTML -fragment
	$ReportOutput += "</TD></TR>"
}
$ReportOutput += "</TABLE>"
$ReportOutput += "</TD></TR></TABLE>"

$Body = ConvertTo-HTML -head $Head -body "$ReportOutput"

$SmtpClient = New-Object system.net.mail.smtpClient
$MailMessage = New-Object system.net.mail.mailmessage
$SmtpClient.Host = "smtp.definit.co.uk"
$mailmessage.from = "scom.report@definit.co.uk"
$mailmessage.To.add("<a href="mailto:sam@definit.co.uk">sam@definit.co.uk</a>")
#$mailmessage.To.add("<a href="mailto:another@definit.co.uk">another@definit.co.uk</a>")
$mailmessage.Subject = "SCOM Daily Healthcheck Report"
$MailMessage.IsBodyHtml = 1
$mailmessage.Body = $Body
$smtpclient.Send($mailmessage)
}

Download the full PowerShell file here:Get-HealthCheck.v2.ps1

Microsoft gives us 2 tools for doing this, Takeown.exe and ICACLs.exe - but there is a catch. To use takeown.exe to to take ownership of a tree you can use the /R recurse option, but you then have to specify a default answer - yes or no. The question that is asked is: "you do not have permission to take ownership, do you want to?" This will strip out existing permissions!!! If you answer yes, you delete the permissions you wish to preserve. If you answer no you do not take ownership of the folder. Without the recurse option you can take ownership of an individual file or folder, but of course this needs to be run as many times as there are folders to be sure.

Engaging with Microsoft product support for this gave us a solution of sorts - namely to run a looped batch file until all the folders were owned, and then running ICACLS.exe. There would be no way of knowing whether this had completed the tree unless you knew already the number of files and folders within the tree - and if you could get that info you would already have permissions! Not a great solution.

Fortunately, using PowerShell and Takeown.exe, we can work around this limitation, by exploiting Get-ChildItem in recursive mode, and changing the default error action to SilentlyContinue. Get-ChildItem will throw an error if it tries to access a folder you do not have permissions to, and you can catch this error and pass the folder to Takeown.exe to seize ownership. Trapping this error does not work because it's a "stop" error - but changing the error action means that the $error variable is populated with the exception details. This is the basis of the first function in my script.

Function: Test-Folder

function Test-Folder($FolderToTest){
$error.Clear()
$ErrorArray = @()
Get-ChildItem $FolderToTest -Recurse -ErrorAction SilentlyContinue | Select FullName
if ($error) {
$ErrorArray = $error + $ErrorArray
foreach ($err in $ErrorArray) {
if($err.FullyQualifiedErrorId -eq "DirUnauthorizedAccessError,Microsoft.PowerShell.Commands.GetChildItemCommand") {
    Write-Host Unable to access $err.TargetObject -Fore Red
    Write-Host Attempting to take ownership of $err.TargetObject -Fore Yellow
    Take-Ownership($err.TargetObject)
    Test-Folder($err.TargetObject)
   }
  }
 }
}

* This section has been updated as per David's comments and solution below - thanks to David for his work!

It's fairly simple code, running through it we clear any existing errors using $error.Clear(), recurse through the folder structure using Get-ChildItem with the errors suppressed. The select statement is for logging output - the full path of the file or folder that we DO have access to. We then check if $error is set - if it is then we had errors accessing a file or folder, so we loop through them and check if the error Fully Qualified ID matches the access denied error message. If it does we write something for the logging and call the Take-Ownership function, followed by looping back with a call to itself to re-test the folder we are working on (and anything below it).

Function: Take-Ownership

  The Take-Ownership function simply calls Takeown.exe against the folder it is passed, then adds entries to the ACL for that folder.

function Take-Ownership {
 param(
  [String]$Folder
 )
 takeown.exe /A /F $Folder
 $CurrentACL = Get-Acl $Folder
 write-host ...Adding NT Authority\SYSTEM to $Folder -Fore Yellow
 $SystemACLPermission = "NT AUTHORITY\SYSTEM","FullControl","ContainerInherit,ObjectInherit","None","Allow"
 $SystemAccessRule = new-object System.Security.AccessControl.FileSystemAccessRule $SystemACLPermission
 $CurrentACL.AddAccessRule($SystemAccessRule)
 write-host ...Adding Infrastructure Services to $Folder -Fore Yellow
 $AdminACLPermission = "DEFINIT\AdminGroup","FullControl","ContainerInherit,ObjectInherit","None","Allow"
 $SystemAccessRule = new-object System.Security.AccessControl.FileSystemAccessRule $AdminACLPermission
 $CurrentACL.AddAccessRule($SystemAccessRule)
 Set-Acl -Path $Folder -AclObject $CurrentACL
}

Running through, it takes the folder name as a parameter, runs Takeown.exe against it using the /A option to add the administrators group. We then get the current ACL from the folder, build a new ACL permission as a string and use that to create a FileSystemAccessRule object. That object is then added to the ACL we copied from the folder. In this example I am adding two accounts, NT AUTHORITY\SYSTEM and a domian group DEFINIT\AdminGroup. The Set-ACL command applies the ACL list with the two new entries to the folder.

The final script

Finally, we can build the script together taking a parameter for the target folder, and a log file. The Functions are called against the target folder - first taking ownership of the root, and then testing the contents. I start and stop a transcript around them to capture output, which can be crucial for proving that you have access!

Start-Transcript $Log
Take-OwnerShip ($RootPath)
Test-Folder($RootPath)
Stop-Transcript

 Download the whole script as a zip file here: set-ownerrecursively

I've been working with a Microsft SCOM PFE (Premier Field Engineer) for the last few months and part of the engagement is an environment health check for the SCOM setup. Based on this Microsoft recommend a series of health checks to for the environment that should be carried out every day. This is summarised as the following:

1. Check the health of all Management Servers and Gateways
2. Check the RMS is not in maintenance mode
3. Review Outstanding Alerts
4. Review Agent's Health Status
5. Review Backup Status
6. Review any Management Group Alerts
7. Review the Pending Management status
8. Review Database Sizes (Operations, Data warehouse, ACS)
9. Review Volume of Alerts
10. Review Alert Latency
11. Document any changes 

From this, there are certain aspects that can't be automated so easily, or shouldn't be - e.g:

3. Review Outstanding Alerts: There is no point in scripting a capture of this as they need to be dealt with in the Console.

5. Review Backup Status: Backups are generally monitored elsewhere and by other software, you could query SQL to find the last backup of the DBs if required.

8. Review Database Sizes: This one I don't agree with being a daily check - if you have sized and tuned your environment correctly then these can be checked on a weekly or monthly schedule.

11. Document any changes: Clearly this isn't going to work as a scripted healthcheck!

Check the health of all Management Servers and Gateways

$ReportOutput = "<H2>Management Servers not in Healthy States</H2>"
$Count = Get-ManagementServer | where {$_.HealthState -ne "Success"} | Measure-Object
if($Count.Count -gt 0) {
 $ReportOutput += Get-ManagementServer | where {$_.HealthState -ne "Success"} | select Name,HealthState,IsRootManagementServer,IsGateway | ConvertTo-HTML -fragment
} else {
 $ReportOutput += "<p>All management servers are in healthy state.</p>"
}

Check the RMS is not in maintenance mode

$RMS = Get-ManagementServer | where {$_.IsRootManagementServer -eq $True}
$criteria = new-object Microsoft.EnterpriseManagement.Monitoring.MonitoringObjectGenericCriteria("InMaintenanceMode=1")
$objectsInMM = (Get-ManagementGroupConnection).ManagementGroup.GetPartialMonitoringObjects($criteria)
$is = "is not"
foreach ($MM in $objectsInMM){
 if($MM.Displayname -eq $RMS.Name){
   $is = "is"
 }
}
$ReportOutput += "<h2>RMS in Maintenance Mode</h2><p>"+ $RMS.Name +" "+$is+" in maintenance mode</p>"

Review Agent's Health Status

$ReportOutput += "<h2>Agents where Health State is not Green</h2>"
$ReportOutput += Get-Agent | where {$_.HealthState -ne "Success"} | select Name,HealthState | ConvertTo-HTML -fragment$ReportOutput += "<h2>Agents where the Monitoring Class is not available</h2>"
$AgentMonitoringClass = get-monitoringclass -name "Microsoft.SystemCenter.Agent"
$ReportOutput += Get-MonitoringObject -monitoringclass:$AgentMonitoringClass | where {$_.IsAvailable -eq $false} | select DisplayName | ConvertTo-HTML -fragment

Review any Management Group Alerts

$ManagementServers = Get-ManagementServer
foreach ($ManagementServer in $ManagementServers){
 $ReportOutput += "<h3>Alerts on " + $ManagementServer.ComputerName + "</h3>"
 $ReportOutput += get-alert -Criteria ("NetbiosComputerName = '" + $ManagementServer.ComputerName + "'") | where {$_.ResolutionState -ne '255' -and $_.MonitoringObjectFullName -Match 'Microsoft.SystemCenter'} | select TimeRaised,Name,Description,Severity | ConvertTo-HTML -fragment
}

Review the Pending Management status

$ReportOutput += "<h2>Agents in Pending State</h2>"
$ReportOutput += Get-AgentPendingAction | sort AgentPendingActionType | select AgentName,ManagementServerName,AgentPendingActionType | ConvertTo-HTML -fragment

Review Volume of Alerts

$ReportOutput += "<h2>Top 10 Repeating Alerts</h2>"
$ReportOutput += get-alert -Criteria 'ResolutionState < "255"' | Sort -desc RepeatCount | select-object –first 10 Name, RepeatCount, MonitoringObjectPath, Description | ConvertTo-HTML -fragment$ReportOutput += "<h2>Agents in Pending State</h2>"
$ReportOutput += Get-AgentPendingAction | sort AgentPendingActionType | select AgentName,ManagementServerName,AgentPendingActionType | ConvertTo-HTML -fragment

The Final Product

My full Daily Health Check Script, with some formatting, output for console and sending an email:

$Head = "<style>"
$Head +="BODY{background-color:#CCCCCC;font-family:Verdana,sans-serif; font-size: x-small;}"
$Head +="TABLE{border-width: 1px;border-style: solid;border-color: black;border-collapse: collapse; width: 100%;}"
$Head +="TH{border-width: 1px;padding: 0px;border-style: solid;border-color: black;background-color:green;color:white;padding: 5px; font-weight: bold;text-align:left;}"
$Head +="TD{border-width: 1px;padding: 0px;border-style: solid;border-color: black;background-color:#F0F0F0; padding: 2px;}"
$Head +="</style>"write-host "Getting Management Health Server States" -ForegroundColor Yellow
$ReportOutput = "<H2>Management Servers not in Healthy States</H2>"
$Count = Get-ManagementServer | where {$_.HealthState -ne "Success"} | Measure-Object
if($Count.Count -gt 0) {
 $ReportOutput += Get-ManagementServer | where {$_.HealthState -ne "Success"} | select Name,HealthState,IsRootManagementServer,IsGateway | ConvertTo-HTML -fragment
} else {
 $ReportOutput += "<p>All management servers are in healthy state.</p>"
}
write-host "Getting RMS Maintenance Mode" -ForegroundColor Yellow
$RMS = Get-ManagementServer | where {$_.IsRootManagementServer -eq $True}
$criteria = new-object Microsoft.EnterpriseManagement.Monitoring.MonitoringObjectGenericCriteria("InMaintenanceMode=1")
$objectsInMM = (Get-ManagementGroupConnection).ManagementGroup.GetPartialMonitoringObjects($criteria)
$is = "is not"
foreach ($MM in $objectsInMM){
 if($MM.Displayname -eq $RMS.Name){
   $is = "is"
 }
}
$ReportOutput += "<h2>RMS in Maintenance Mode</h2><p>"+ $RMS.Name +" "+$is+" in maintenance mode</p>"write-host "Getting Agent Health Status" -ForegroundColor Yellow
$ReportOutput += "<h2>Agents where Health State is not Green</h2>"
$ReportOutput += Get-Agent | where {$_.HealthState -ne "Success"} | select Name,HealthState | ConvertTo-HTML -fragment$ReportOutput += "<h2>Agents where the Monitoring Class is not available</h2>"
$AgentMonitoringClass = get-monitoringclass -name "Microsoft.SystemCenter.Agent"
$ReportOutput += Get-MonitoringObject -monitoringclass:$AgentMonitoringClass | where {$_.IsAvailable -eq $false} | select DisplayName | ConvertTo-HTML -fragmentwrite-host "Getting Management Server Alerts" -ForegroundColor Yellow
$ReportOutput += "<h2>Management Server Alerts</h2>"
$ManagementServers = Get-ManagementServer
foreach ($ManagementServer in $ManagementServers){
 $ReportOutput += "<h3>Alerts on " + $ManagementServer.ComputerName + "</h3>"
 $ReportOutput += get-alert -Criteria ("NetbiosComputerName = '" + $ManagementServer.ComputerName + "'") | where {$_.ResolutionState -ne '255' -and $_.MonitoringObjectFullName -Match 'Microsoft.SystemCenter'} | select TimeRaised,Name,Description,Severity | ConvertTo-HTML -fragment
}write-host "Getting Top 10 Unresolved Alerts" -ForegroundColor Yellow
$ReportOutput += "<h2>Top 10 Unresolved Alerts</h2>"
$ReportOutput += get-alert -Criteria 'ResolutionState < "255"'  | Group-Object Name | Sort-object Count -desc | select-Object -first 10 Count, Name | ConvertTo-HTML -fragmentwrite-host "Getting Top 10 Repeating Alerts" -ForegroundColor Yellow
$ReportOutput += "<h2>Top 10 Repeating Alerts</h2>"
$ReportOutput += get-alert -Criteria 'ResolutionState < "255"' | Sort -desc RepeatCount | select-object –first 10 Name, RepeatCount, MonitoringObjectPath, Description | ConvertTo-HTML -fragmentwrite-host "Getting Agents in Pending State" -ForegroundColor Yellow
$ReportOutput += "<h2>Agents in Pending State</h2>"
$ReportOutput += Get-AgentPendingAction | sort AgentPendingActionType | select AgentName,ManagementServerName,AgentPendingActionType | ConvertTo-HTML -fragmentwrite-host "Getting Overrides in Default Management Pack" -ForegroundColor Yellow
$ReportOutput += "<h2>Overrides in Default Management Pack</h2>"$OverrideCount = Get-ManagementPack | where {$_.DisplayName -match "Default Management Pack"} | get-override | measure-objectif($OverrideCount.Count -gt 2){
 foreach ($monitor in Get-ManagementPack | where {$_.DisplayName -match "Default Management Pack"} | get-override | where {$_.monitor}) {
  $ReportOutput += get-monitor | where {$_.Id -eq $monitor.monitor.id} | select-object DisplayName,Description | ConvertTo-HTML -fragment
  $ReportOutput += "<br />"
 }
 foreach ($rule in Get-ManagementPack | where {$_.DisplayName -match "Default Management Pack"} | get-override | where {$_.rule}) {
  $ReportOutput += get-rule | where {$_.Id -eq $rule.rule.id} | select-object DisplayName,Description | ConvertTo-HTML -fragment
  $ReportOutput += "<br />"
 }
} else {
 $ReportOutput += "<p>There are no unexpected overrides in the Default Management Pack</p>"
}$Body = ConvertTo-HTML -head $Head -body "$ReportOutput"$SmtpClient = New-Object system.net.mail.smtpClient
$MailMessage = New-Object system.net.mail.mailmessage
$SmtpClient.Host = "smtp.definit.co.uk"
$mailmessage.from = "scom.report@definit.co.uk"
$mailmessage.To.add("<a href="mailto:sam@definit.co.uk">sam@definit.co.uk</a>")
#$mailmessage.To.add("<a href="mailto:another@definit.co.uk">another@definit.co.uk</a>")
$mailmessage.Subject = "SCOM Daily Healthcheck Report"
$MailMessage.IsBodyHtml = 1
$mailmessage.Body = $Body$smtpclient.Send($mailmessage)

You can download the file here (zip): Get-HealthCheck

This post is nothing more than a shameless request for sponsorship! As the title suggests, I am running the London marathon this year (in 96 days!) for the charity "The Lighthouse Group". Check out the TLG site for more detail on what they do, but in a nutshell they are a charity that works with young people who have been excluded from school, at risk of exclusion or are at crisis point in their education. It's a really worthwhile cause and my father-in-law has just been involved in opening a TLG center based in Normanton, Yorkshire

I'd appreciate any contribution, big or small! It's fair to say I'm not quite the right build to run a marathon, so a little bit of sponsorship would be very encouraging! I've been training since late August last year, and am currently managing two 7 mile runs a week, plus a game of football and a couple of swims! Keep up to date with my progress over on my Runkeeper profile.

The State Change event details are as follows:

Date and Time: 05/12/2011 12:44:00
Property Name Property Value
State Error
Message Unable to obtain Test-MAPIConnectivity result - The mailbox database 'MAILBOXSERVER01\Recovery Storage Group\Database01' is a recovery mailbox database. Only non-recovery mailbox databases are monitored by this task.

Fortunately we can create a dynamic group that will populate any Mailbox Database objects containing the "Recovery Storage Group" string in the display name - any new recovery databases will be automatically populated as they are discovered. We can override this monitor for the group, and forget about this issue!

Open the Authoring console and select the Groups tab. Select Create a New Group from the Actions pane.

Enter a descriptive name for the group - Exchange 2007 Recovery Storage Groups, and select the Management Pack to store it in - not the Default MP!

Click next for the Explicit Members, and then click "Create/Edit rules..." on the Dynamic Members tab. Find "Exchange 2007 Mailbox Database" in the dropdown and add it to the rule conditions. Select the following conditions: "Display Name", "Contains" and then "Recovery Storage Group".

Finish the wizard to create the group. Click on the Monitors tab and locate the "Exchange 2007 Test MAPI Connectivity Monitor", right click and open the properties. Select the Overrides tab.

Select "Override..." and then "For a group...", then find the group you just created.

Override the "Enabled" parameter and select "False" to disable the Monitor, and click OK to apply the override.

After a few minutes, the override will take effect and reset the health of any Mailbox roles with a Recovery Storage Group.

Like many organisations there is a root CA (we'll call it ROOTCA01), and then a subordinate CA (we'll call that SUBCA01). OPSMGM01 has a certificate to identify itself and has certificates for ROOTCA01 and SUBCA01 in it's Trusted Root Certificate Authorities.

The certificate to secure the connection between OpsMgr Gateway (OPSGW01) and the OpsMgr Management Server (OPSMGM01) is issued by SUBCA01 and is installed on OPSGW01, and to validate the certificate chain SUBCA01's certificate is also installed in the Trusted Root Certification Authorities. Opening OPSGW01's certificate and examining the Certificate Path tab shows the certificate is valid all the way up to the issuing CA - SUBCA01.

The connection will not work - OPSGW01 logs the following events:

Log Name:      Operations Manager
Source:        OpsMgr Connector
Date:          05/01/2012 10:18:28
Event ID:      21016
Level:         Error
Computer:      opsgw01.definit.co.uk
Description:   OpsMgr was unable to set up a communications channel to opsmgm01.definit.co.uk and there are no failover hosts.  Communication will resume when opsmgm01.definit.co.uk is available and communication from this computer is allowed.

Log Name:      Operations Manager
Source:        OpsMgr Connector
Date:          05/01/2012 10:18:25
Event ID:      20070
Level:         Error
Computer:      opsgw01.definit.co.uk
Description:   The OpsMgr Connector connected to opsmgm01.definit.co.uk, but the connection was closed immediately after authentication occurred.  The most likely cause of this error is that the agent is not authorized to communicate with the server, or the server has not received configuration.  Check the event log on the server for the presence of 20000 events, indicating that agents which are not approved are attempting to connect.

Log Name:      Operations Manager
Source:        OpsMgr Connector
Date:          05/01/2012 10:18:24
Event ID:      21002
Level:         Warning
Computer:      opsgw01.definit.co.uk
Description:   The OpsMgr Connector could not accept a connection from xxx.xxx.xxx.xxx:5723 because mutual authentication failed.

Log Name:      Operations Manager
Source:        OpsMgr Connector
Date:          05/01/2012 10:18:24
Event ID:      20067
Level:         Warning
Computer:      opsgw01.definit.co.uk
Description:   A device at IP xxx.xxx.xxx.xxx:5723 attempted to connect but the certificate presented by the device was invalid.  The connection from the device has been rejected.  The failure code on the certificate was 0x800B0109 (A certificate chain processed, but terminated in a root certificate which is not trusted by the trust provider.).

It's the last event that led me to check the certificate chain for the SUBCA01 certificate, which was installed and trusted but did not validate up the chain to ROOTCA01. Installing the ROOTCA01 certificate resolved this issue.

param( [string] $vCenterServer = $(Read-Host -prompt "Enter vCenter Server Name"),
[string] $TargetPolicy = $(Read-Host -Prompt "Enter target policy (RoundRobin, Fixed or MostRecentlyUsed)"),
[string] $TargetHost = $(Read-Host -Prompt "Enter target Host"),
[switch] $WhatIf)

# Add the VI-Snapin if it isn't loaded already
if ((Get-PSSnapin -Name "VMware.VimAutomation.Core" -ErrorAction SilentlyContinue) -eq $null ) {Add-PSSnapin -Name "VMware.VimAutomation.Core"}

Connect-VIServer $vCenterServer | out-null

Write-Host "Connected to: " $vCenterServer -ForegroundColor Green
Write-Host "Target PSP: " $TargetPolicy -ForegroundColor Yellow
Write-Host

switch ($TargetPolicy) {
RoundRobin { $DisplayPolicy = "VMW_PSP_RR"; }
MostRecentlyUsed { $DisplayPolicy = "VMW_PSP_MRU"; }
Fixed { $DisplayPolicy = "VMW_PSP_FIXED"; }
default { Write-Warning "Unknown PSP selected! Please consult the help and try again."; exit }
}

Write-Host "Setting Policy to"$TargetPolicy" on "$TargetHost -ForegroundColor Green

if($WhatIf) {
$vHost = Get-VMHost -Name $TargetHost
$vHost | Set-VMHost -State Maintenance -Evacuate -WhatIf
$vHost | Get-ScsiLun -LunType "disk" -ErrorAction SilentlyContinue | where {$_.IsLocal -eq $false -and $_.MultipathPolicy -ne $TargetPolicy} | Set-ScsiLun -MultipathPolicy $TargetPolicy -WhatIf
$vHost | Set-VMHost -State Connected -WhatIf
} else {
$vHost = Get-VMHost -Name $TargetHost
Write-Host "Setting "$TargetHost" to Maintenance Mode" -ForegroundColor White
$vHost | Set-VMHost -State Maintenance -Evacuate
$vHost | Get-ScsiLun -LunType "disk" -ErrorAction SilentlyContinue | where {$_.IsLocal -eq $false -and $_.MultipathPolicy -ne $TargetPolicy} | Set-ScsiLun -MultipathPolicy $TargetPolicy
Write-Host "Exiting Maintenance mode on"$TargetHost -ForegroundColor White
$vHost | Set-VMHost -State Connected
}

The easy (and wrong) option here is to go with the less secure option and distribute a RunAs account to ALL servers. There are lots of reasons why you wouldn’t want to distribute the credentials to every server in your SCOM installation – but just from a security standpoint, you shouldn’t do it! Selecting the “More Secure” option and distributing credentials only to servers which will require them is a much safer bet.

You can view the members of the DFS discovered inventory in the SCOM Console by going to the “Discovered Inventory” view and changing the target type to “Replication Member” – which is great: you can see all the Servers involved in the DFS replication topology. But there’s no easy way to add these to a RunAs credential to distribute.

To narrow it down to a short list, you can open a Operation Manager Shell prompt and  list any monitoring classes which have “DFS” in the name – there are about 6 or so:

Get-MonitoringClass | where {$_.Name –match “DFS”}

The one that matches my SCOM console view is “Microsoft.Windows.DfsReplication.ReplicationGroupMember” so I want to select all the monitoring-objects that match this discovery and export the “Path” (server name) to a csv file:

Get-MonitoringClass | where {$_.Name –match “Microsoft.Windows.DfsReplication.ReplicationGroupMember”} | get-monitoringobject | select-object Path | export-csv c:\DFS-Members.csv

I’ve not yet figured out how to add these to the RunAs account credential distribution via PowerShell, so I’m afraid it’s a manual process from here. To make it easier I opened the csv in Excel and filtered out duplicates (for servers with multiple DFS shares) before pasting the servers in individually to the distribution dialogue.

Once the RunAs account has been downloaded by the Agents, and if you've added it correctly to your "DFS Replication Monitoring Account" profile, you should start to see the Backlog Monitoring view beginning to populate.

Get the NICs right first

In this case I came to a project after the initial installation of the array and there was no dedicated intra-array network installed. I added a new NIC to each VM and configured the IP addressing, VLANs and routing, but could not get the intra-array network to ping, let alone talk to each other. So the lesson here is to set up the servers with their NICs before you install TMG - Microsoft recommend a dedicated intra-array network and every bit of experience I have with TMG arrays confirms that.

Get the NIC Binding order right

This is simple, the order I have found to work is:

• Intra-array Network
• Private/Internal Network
• Public/External Network

Some people recommend the Private/Internal network first, then the Intra-array, but I have found that this order works better (anyone able to dispute this or give me a reason why it should be the other way?). The key thing is that the External Network (which should be your default Gateway) is last in the binding order, which brings me to the next point...

Get the gateway and routing right

• Default Gateway: The only NIC with a Default Gateway set should be the Public/External NIC
• DNS: The only NIC with DNS configured should be your Private/Internal NIC
• Register in DNS: The only NIC registering in DNS should be the Private/Internal NIC
• Client for Microsoft Networks: Only enabled on the Private/Internal NIC
• File and Print Sharing for Microsoft Networks: Only enabled on the Private/Internal NIC
• NetBIOS over TCP/IP: Only enabled on the Private/Internal NIC

Add any static and persistant routes required and make sure you can access those networks before installing TMG. This allows you to get the routing right without the complication of TMG rules and firewalls.

Then, and only then, install TMG

I’ve recently installed OTRS on a Windows Server 2008 R2 (64-bit) server, including experimenting with various combinations of IIS/Apache, MSSQL2008/MySQL, ActiveState Perl 32-bit/64-bit, different configurations and setups - these are my findings:

• IIS7, MSSQL (64), ActiveState Perl (64) – to make use of the native IIS7 webserver and 64-bit Perl. The server does run but performs abysmally, and you have to force IIS to run a 32-bit application pool to get Perl to work.
• Apache2.2 (32), MSSQL (64) and ActiveState Perl (32) – again OTRS will run but performance is grim
• Apache2.2 (64 unofficial binaries), MySQL (64) and ActiveState Perl (64) – this seemed the most promising approach but without a 64-bit version of mod_perl the performance was worse than the final combo
• Apache2.2 (32), MySQL (64) and ActiveState Perl (32) – this performed the best, and although there are slow portions (SysConfig) the general user experience was good.

None of these combinations came close to the performance of OTRS running on a native Linux server, my 64-bit Ubuntu server absolutely flew, with less processor and RAM than the Windows box. In short, if you have the skills, use the Linux option. Yes, yes I do feel a little dirty now, sorry Mr Gates.

So, the final setup I have opted for is:

• A Virtual Machine running Windows Server 2008, 2GB RAM and 2 vCPUs at 3.2Ghz
• MySQL Server 5.5, 64-bit
• Apache 2.2, 32-bit
• ActiveState Perl, 32-bit

Installing Apache HTTP Server

Make sure you don’t have IIS running (eye-roll), and then click through the wizard…

Configure the network and server details

Do a custom installation and ensure that you install it to D:\Apache\

Apache should now be running on the local machine, navigate to http://localhost to check that it’s running. You should see a page saying “It works!”

Installing MySQL

Installing MySQL is again a very simple click-through process, once you’ve downloaded it from http://dev.mysql.com/downloads/. I am installing the 64-bit server to make the most of the 64-bit performance on this server.

Changes to MySQL’s default configuration file

Stop the MySQL service and edit c:\Program Files\MySQL\MySQL Server 5.5\my.ini in your favourite text editor– you will need to run it as an Administrator.

Change the default data directory (you wouldn’t leave a MSSQL DB on the System drive would you?!) – create a folder on the data drive (I use D:\MySQL\Data).

#Path to the database root
datadir="D:\MySQL\Data"

Go to c:\ProgramData\MySQL\MySQL 5.5\data\ and move all the files and folders there to the folder you specified for datadir.

For performance, it’s also recommended that you change the query_cache_size to “32M”

You can now start the MySQL service again.

Installing ActiveState Perl

You can download the 32-bit version of ActiveState Perl here: http://www.activestate.com/activeperl/downloads

Run through the installation, accepting the defaults apart from changing the location of the installation to D:\Perl\:

Installing OTRS

At this point, we will download and extract the OTRS installation from http://ftp.otrs.org/pub/otrs/otrs-3.0.9.zip and extract it to D:\OTRS\

Once that’s done, we can begin joining up the dots.

Installing the correct Perl Packages

Run a command prompt as administrator and navigate to the OTRS installation directory (for me, d:\OTRS).

perl bin\otrs.CheckModules.pl

You can then use the Perl Package Manager to install the missing required packages, and some of the options:

ppm install date::format

ppm install dbd::mysql

ppm install json::xs

ppm install apache::reload

ppm install net::dns

ppm install net::ldap

ppm install pdf::api2

ppm install http://cpan.uwinnipeg.ca/PPMPackages/12xx/mod_perl.ppd

(this one requires that you point it to the correct Apache modules folder, for me d:\Apache\modules)

If you re-run the otrs.CheckModules.pl script it should now pass all modules.

Configuring Apache

The main Apache configuration file is installed at d:\Apache\conf\httpd.conf, open it in Notepad and add the following lines at the end of the config file:

# load mod_perl
LoadFile 'd:/Perl/bin/perl512.dll' #match your perl installation
LoadModule perl_module modules/mod_perl.so

Include 'd:/OTRS/scripts/apache2-httpd.include.conf' # match your OTRS installation

Also modify the  DirectoryIndex directive to be index.pl rather than index.pl.

DirectoryIndex index.pl

Configuring OTRS

Configuring OTRS is initially a case of changing paths referenced in the configuration files to the Windows paths that match your environments.

D:/OTRS/scripts/apache2-httpd.include.conf

I performed a search/replace for “/opt” with “D:”, and checked through after.

D:\OTRS\scripts\apache2-perl-startup.pl

D:\OTRS\kernel\Config.pm

Rename D:\OTRS\kernel\Config.pm.dist to Config.pm, and open Config.pm in your text editor.

Add the following lines to configure logging:

$Self->{'LogModule'} = 'Kernel::System::Log::File';
$Self->{'LogModule::LogFile'} = "$Self->{Home}/var/log/otrs.log";

Running the web installer

Run http://localhost/otrs/installer.pl in your browser of choice and click through the initial details and accepth the licensing agreement:

Enter the MySQL root user and password you created earlier, and test the settings

Next configure a user, password, database name and create the database. you don’t have to change the default names, but if you don’t change the password from the default “hot” to something complex, you deserve to get hacked!

Configure the general system, FQDN, admin email address and logging

Configure incoming and outbound email

And finish!

You can now log in to your OTRS installation – but we’re not finished yet!

Configuring Scheduled Jobs

The MSI installer uses Cron4Win, but in my experience it was barely working, clunky and not at all suitable for a production service. Since the jobs are just .pl scripts, it’s better to run them from the Scheduled Tasks.

Open up the Task Scheduler and run the Create Task action (not Create Basic Task). Name the task and configure the user you want to run under. Ensure that you select to run whether the user is logged in or not, and run with elevated privileges. I have used a specific account with locked down privileges.

Move on to the “Triggers” panel and create a schedule for the task (see the table below for my config)

Move to the “Actions” panel and create an action to start a program and pass the script as an argument (again, see the table below for my config).

You can accept the default settings for the rest of the panes.

Run each of the jobs manually to ensure that they process OK, without any errors. I have exported my jobs as XML files, which are available here to download and import (assuming you’ve put all the files in the same place as me!)

That’s it, OTRS is ready for you to configure, which is way beyond the scope of this post!

Existing Network layout and Design

Assuming there is a core network switch and that it is a Layer 3 enabled switch which has inter-VLAN routing configured. By default all the VLANs can talk to each other, routed through the switch. The switch also is configured with a gateway of last resort pointing to the firewall’s internal IP – this allows internet access.

So the headlines are:

• Existing Production VLANs – VLAN10 iSCSI (10.1.10.0/24), 11 Server (10.1.11.0/24) and 12 Client(10.1.12.0/24) – these all route through the core switch and can see each other.
• Create a Guest VLAN – to be created VLAN13 (10.1.13.0/24), which can access the internet, but not the existing VLANs
• Create a Secure Wireless LAN – all traffic assigned to VLAN12. Since it’s a domain environment this will use PEAP authentication, so clients can use their domain password to access the WLAN.
• Create a Guest Wireless LAN – all traffic assigned to VLAN13. This will use a static WPA2 access passphrase which can be changed regularly – since it won’t be used by domain clients or those who will repeatedly access it, it's not a huge admin overhead.

I’m not going to cover setting up a Domain, Certificate Authority, or Internet Authentication Service. I am assuming you have this already, and have issued a Server certificate to your IAS server, and the CA is trusted throughout your domain clients. My demo lab set up:

• Wireless Access Point (Cisco WAP4410N) – Definit-WAP
• Core Network Switch (Cisco 3750) – Definit-SW
• Active Directory Domain Controller – DefinIT-DC
• Public Key Infrastructure – DefinIT-CA
• Windows Server 2003 IAS – DefinIT-IAS

Preparing the Network Core

All commands are from the Configure prompt (config t)

The first task is to configure the new Guest VLAN13 using the commands below. The IP address assigned to the vlan13 interface acts as the gateway for VLAN13 on the core switch. (VTP will propagate the VLAN settings to the access switches, if configured).

vlan1
name DefinITGuest
interface Vlan13
description DefinIT Guest VLAN
ip address 10.1.13.1 255.255.255.0

Next configure the interface for the Cisco WAP4410N to plug into on the core switch (in this case G2/0/13) – port needs to be in trunk mode to carry multiple VLANs. I’ve configured the default VLAN to be the Server VLAN11 as this is the one I want the web management interface to be accessible on. Any untagged traffic will be assigned to this by default. Here you could also restrict the allowed VLANs using “switchport trunk allowed 13” but this wouldn’t allow the secure WLAN to access the server/client VLANs.

interface GigabitEthernet2/0/13
description DefinIT-WAP Trunk
switchport trunk encapsulation dot1q
switchport trunk native vlan 11
switchport mode trunk
no shutdown

The Guest WLAN will not be allowed access to the server VLAN in any form, so it can’t use the client DHCP server. Fortunately the switch is more than capable of handling that – we move on to the DHCP pool configuration. Because the DHCP scope is on the same IP network as the VLAN13 interface, only that interface that will respond to DHCP requests (which is good, because I don’t want my network ruined by fighting DHCP servers!)

ip dhcp pool GuestWLAN
network 10.1.13.0 255.255.255.0
default-router 10.1.13.1
dns-server 8.8.8.8, 8.8.4.4

Now that the plumbing is set up, we need to control who is allowed to access what. This means creating an Access List to deny the guest VLAN access to the production VLANs. Note that the format for the ACL does not use a subnet mask, but a wildcard mask. You need to subtract each octet of your subnet mask from 255 to get the wildcard mask (e.g 255.255.255.0 becomes 255-255=0, 255-255=0, 255-255=0 and 255-0=255 to get 0.0.0.255).

ip access-list extended DefinIT_GUEST
remark Deny Guest VLAN13 access to other VLANs
deny ip any 10.1.10.0 0.0.0.255
deny ip any 10.1.11.0 0.0.0.255
deny ip any 10.1.12.0 0.0.0.255
permit ip any any

Finally, apply the Access List to the Guest VLAN13 interface. Note that the direction is “in” which seems counter-intuitive but is correct. The perspective is from the switch, so traffic is coming in from a client on the guest VLAN to the VLAN13 interface on the switch.

interface Vlan13
ip access-group DefinIT_GUEST in

That’s it, core network configured!

Configure IAS to provide RADIUS authentication

Create a new RADIUS client by selecting the RADIUS Clients folder and right-click – new. Configure a friendly name for the Wireless Access Point, and the IP you’re using for the WAP (for me, DefinIT-WAP and 10.1.13.20). Configure the Client-Vendor to Cisco – or the vendor you’re using, and a strong shared secret (random, 13 characters upper/lower/alpha/numeric/special will do nicely).

Create an new Remote Access Policy using the Remote Access Policy wizard (right-click Remote Access Policies and select new…):

Select Wireless as the access method, and select a Windows Security group to allow access:

Select PEAP as the Authentication method, and configure the server certificate for identification, and the EAP type to use MSCHAP-v2. If you have issued client certificates to all users, you can add Smart Card or other Certificate to the EAP authentication methods.

Configuring the Access Point

I’m assuming you can manage to turn the thing on, access it’s web interface and assign the static IP you picked earlier to the AP, now we can configure the authentication and VLANs for the guest and secure WLANs.

Open the Wireless > Basic settings and configure your two SSIDs, save and then open the Security page.

Here you can configure WPA2-Personal for the Guest SSID, and WPA2-Enterprise Mixed for the Secure SSID. Configure the IP address of you RADIUS server and the Share Secret you configured earlier.

Now move onto the VLAN and QoS page – here you need to enable VLAN but leave the defaults otherwise. Under QoS you need to assign the VLAN ID for each network – 13 for Guest and 12 for Secure

That’s more or less it, time to test with a handy wireless client sitting nearby…

However I am often left wondering how we can maintain a balance. I have all to often seen IT professionals falling into the trap chasing the latest and greatest and rushing to try to implement or learn new emerging technologies without much thought to what they already have.

SharePoint Foundation 2010 introduces a new 10GB limit with SQL Server Express 2008 R2 for database sizes, and something called “Remote BLOB” storage, which limits you only to your storage. Traditionally, BLOBs (Binary Large OBjects) have been stored in the Database file itself with the structured data. Put simply Remote BLOB stores the file data in a folder rather than the database. This makes a lot of sense to me as the largely simple, unstructured file data does not really require the structured data environment of the database.

Installing SQL Express 2008 R2

For some reason best known to themselves, Microsoft didn’t include the option to install SQL Express 2008 R2 with the SharePoint Foundation installer, presumably the want to keep the installer size down to a minimum. So the first port of call is to download the SQL Express installer from the Microsoft SQL Express download site. Installing is very straight-forward, so I won’t go into much detail. Once downloaded, run the installer and click through the install, I created a user account for SQL and SharePoint services to run under.

Provisioning the FILESTREAM provider

Once installed, we can move on to enabling the FILESTREAM provider. Open the SQL Server Configuration Manager and select the SQL Server Services node. Select the SQL instance you just installed, right click and select “Properties”. Select the FILESTREAM tab and tick all 3 boxes to enable:

Next open SQL Server Management Studio and open a new query window. Run the following query to enable FILESTREAM:

EXEC sp_configure filestream_access_level, 2
RECONFIGURE

At this point, we move on to installing SPF 2010.

Installing SharePoint Foundation 2010

Run the installer and click “Install software prerequisites” – check you’re happy with the listed updates, roles and features and then click next.

The go for a coffee, or two…

This unfortunately requires a restart before we can continue, which is a shame! Once the server has bounced, re-run the installer and click “Install SharePoint Foundation”. At this point I had an issue with the installer telling me “A system restart from a previous installation or update is pending. Restart your computer and run setup to continue.” Restarting did not resolve this issue, I had to modify the registry keys mentioned on this technet blog.

Once I’d done that, I was able to run the installer, choose the location for the search index data and watch the blue bar:

Run the SharePoint Products Configuration Wizard and create a new farm

Enter the database details and create a pass phrase for joining the SPF 2010 farm

Configure the SharePoint Central Administration Web Application (defaults are fine)

After a bit of crunching you should see the “Configuration Successful” page.

Configuring the SharePoint Farm

Once finished the Central Administration web page should open and offer you the chance to configure the farm, either through a wizard or manually – I’m going to go through the wizard. I’m not going to document the wizard on here because it’s pretty straightforward. You do need to create a Site and Content Database in order to enable the BLOB Store.

Provision a BLOB Store with the FILESTREAM provider

Return to SQL Server Management Studio and select the database for the Site you’ve just created and want to enable BLOB storage for and open a new query window. In my case, this is WSS_Content (and will be for you used the wizard to create your first site). Run the following queries against the content DB:

use [ContentDbName]
if not exists (select * from sys.symmetric_keys where name = N'##MS_DatabaseMasterKey##')create master key encryption by password = N'Admin Key Password !2#4'

use [ContentDbName]
if not exists (select groupname from sysfilegroups where groupname=N'RBSFilestreamProvider')alter database [ContentDbName] add filegroup RBSFilestreamProvider contains filestream

use [ContentDbName]
alter database [ContentDbName] add file (name = RBSFilestreamFile, filename = 'c:\Blobstore') to filegroup RBSFilestreamProvider

Installing RBS on the SharePoint Foundation Server

After a lot of head scratching I finally found mention that BEFORE you install RBS, make sure you enable NAMED PIPES for this instance of SQL Server or the RBS won’t install correctly. Head over to the SQL Server 2008 Feature Pack site and download the RBS package for your server type, which comes down as RBS.msi. Run a command prompt as an administrator and then install the MSI using the following command, changing DBNAME, DBINSTANCE to your environment (DBINSTANCE for me was the default instance on the server, so the server name was required):

msiexec /qn /lvx* rbs_install_log.txt /i RBS.msi TRUSTSERVERCERTIFICATE=true FILEGROUP=PRIMARY DBNAME="WSS_Content" DBINSTANCE="<server name>" FILESTREAMFILEGROUP=RBSFilestreamProvider FILESTREAMSTORENAME=FilestreamProvider_1

You won’t get any output from that command, but after a minute or so you should be able to check the log file that there is a line containing “Product: SQL Server 2008 R2 Remote Blob Store -- Installation completed successfully.” You should also be able to see several new “mssqlrbs*” tables in your content database.

Enable and test RBS

Open the SharePoint 2010 Management Shell on server and enter the following commands to test and enable RBS:

$cdb = Get-SPContentDatabase -WebApplication http://SiteName

[No output]

$rbss = $cdb.RemoteBlobStorageSettings

[No output]

$rbss.Installed()

Should respond with “True”

$rbss.Enable()

[No output]

$rbss.SetActiveProviderName($rbss.GetProviderNames()[0])

[No output]

$rbss | fl

Enabled                     : True

ActiveProviderName          : FilestreamProvider_1

MinimumBlobStorageSize      : 0

UpgradedPersistedProperties : {}

As a final test, browse to our SharePoint site and upload a file >100k to the default document library. You can then check the size of the file you uploaded to the files in the BLOB storage folder you created and check they match (they won’t be called the same thing!).

Storing your files as BLOBs in Remote storage has it’s detractors, and I’m not going to get into that now – this article was to show how to do it if you have read around and require to do so. I hope it helps!

Links

• Install and configure RBS with the FILESTREAM provider (SharePoint Foundation 2010)
• Download Microsoft SharePoint Foundation 2010
• SharePoint Foundation 2010… A simple install
• Common SharePoint 2010 Installation Problems
• SQL Server 2008 R2 Feature Pack

SSTP VPN Requirements

• Clients must be Windows 7/Server 2008 or newer
• Certificate – either commercial or an internal Certificate Authority
• Published CRL – SSTP clients check for the Certificate Revocation List of the CA
• If you already have an SSL listener (e.g. for Exchange publishing rules) then you need a dedicated IP address for the SSTP connection

TMG is configured as a “back-firewall” in this environment, with an adaptor in the LAN and one in the Perimeter (DMZ). The DMZ has a NAT relationship to the External public IPs.

Requesting a Certificate from a Certificate Authority from Certificate Services 2003

Create a sstp.inf file to define the request. The subject CN MUST be the exact URL you’re going to use to access the SSL VPN, you can’t use a Subject Alternative Name (SAN):

[NewRequest]

Subject="CN=sstp.definit.co.uk"

Exportable=TRUE

KeyLength=2048

KeySpec=1

MachineKeySet=TRUE

[EnhancedKeyUsageExtension]

OID=1.3.6.1.5.5.7.3.1 ; Server Authentication
OID=1.3.6.1.5.5.7.3.2 ; Client Authentication

[RequestAttributes]
CertificateTemplate = WebServer

Open a command prompt as administrator and generate a request file:

certreq –new c:\sstp.inf c:\sstp.req

This generates a request file, which we’ll use to create the certificate on the Certificate Authority. Normally I would submit the request directly to the CA, but TMG does not play nicely with the RPC protocol, and I’ve not managed to get this working.

Copy the sstp.req file over to your CA and run another command prompt. Submit the request and save the response back to the TMG server:

certreq –submit –PolicyServer “CA01\CA Name” c:\sstp.req c:\sstp.cer

This creates the certificate required for the SSL VPN, which you can then accept back on the TMG server:

certreq –accept c:\sstp.cer

If you need to, also install the CA's Root Certificate in the Trusted Root Certification Authorities.

Configuring VPN Client Access

There are 6 steps in the Remote Access Policy (VPN) panel of the TMG console for setting up VPN connections, not all of them required.

Configure Address Assignment Method and Enable VPN Client Access

So, click the first link and configure the Address Assignment. Since this TMG server has a leg in the LAN, it can use the DHCP server there, select the Internal Network. You can configure a static pool of IPs if you want – e.g. to put the clients on a more secure VLAN or to control access to resources based on a subnet:

Select the Access Networks tab, and you can configure which networks the VPN clients can connect to. I’ve selected External and Perimeter for obivious reasons – that’s where my clients will be coming from. For testing purposes I’ve also selected the Internal network so I can test the SSL VPN from my desktop.

On the Authentication tab, select MS-CHAPv2 – this will allow the most secure username and password authentication method without issuing client certificates or smart cards (EAP). CHAP and PAP are basically a no-go.

I haven’t configured a RADIUS server for this connection to keep it simple. The “Enable VPN Client Access” link will enable the System Policy firewall rule.

Specify Windows Users or select a RADIUS Server

Click on the Specify Windows Users link and select a group from your Active Directory to allow VPN access.

Under the General tab you can enable or disable VPN access and set a limit on the number of users allowed at the same time.

Under protocols, untick the Enable PPTP box and check the Enable SSTP

Click “Select Listener…” and create a New listener to open the Web Listener Wizard. Name the listener

Select the Networks you want to configure the Listener for – the same as you configured for the Remote Access Policy is a good idea! To ensure that the SSL VPN has a unique certificate and IP address, specify the IP address you want the listener to listen on in the Perimeter network:

Select the Certificate you created earlier for the Listener and complete the wizard

Verify VPN Properties and Remote Access Configuration

Essentially, that means check what you’ve just done…not sure why this warrants a step MS?

View Firewall Policy for the VPN Clients Network

Create a rule to control where you want your VPN Clients to access – for me the VPN clients are trusted and therefore I created a rule to allow “All outbound traffic” to “All Networks”:

View Network Rules

Similarly, check that you’re happy with the Network Rules for the VPN clients – I didn’t need to change anything here.

Configure Quarantine (Optional)

As this is optional and I only want a simple setup so this will be a later post, hopefully!

Publishing your Certificate Revocation List

My CA is configured to add an external URL to the Certificates it creates – crl.definit.co.uk. This means that the CRL can be made publicly available, this is a requirement for SSTP clients which by default will check a certificate’s validity.

Using the Web Publishing Rule Wizard I created a rule called “CRL Publishing”  to “allow” access to a single Web site or load balancer using HTTP connections to the internal site name and with a path of /CertEnroll/* on the Public name is the external URL – crl.definit.co.uk. I created a new HTTP only listener, set the rule to “No Authentication” and allowed “All users” access.

Configuring the SSTP client

Configuring the clients is very simple – on Windows 7 and Server 2008 it’s the same. First of all, ensure that you trust the certificate authority that issued the SSTP certificate created earlier, then open the “Set up a Connection or Network” wizard and select “Connect to a workplace”:

Connect via your internet connection and enter the url (sstp.definit.co.uk) and a nice name for it:

Enter your Active Directory User credentials (must be a member of the user group added earlier) and click “Connect”

All being well, you’re connected! If you get errors, double check the certificates!

I am however glad to say finally Microsoft have taken notice of this and produced a very simple and effective deployment toolkit.

The Microsoft Web Platform Installer (now in version 3.0)

I have recently deployed an IIS7.0 server that required PHP and MySQL using this tool and I am very happy with  the results!

I know generally any system admin will avoid "wizards" as it were but in this instance it is time well saved!

Having recently managed several Exchange 2010 migration projects, one of the best new features which really sells it to systems administrators is the Online Archive. “No more managing PST files? When can we have it installed by?”

The problem is, once they’ve purchased licensing for Exchange 2010 and installed and configured the server, migrated the users’ mailboxes and decommissioned the old Exchange 2003 server, the Online Archive feature is not available. The users have been enabled, and as of SP1 we have a separate Archive mailbox database configured on slow (cheap) storage, but the Online Archive is nowhere to be found in Outlook. If the users log on using OWA, lo and behold the Online Archive is available.

Now, fair enough, Microsoft require an Enterprise Client Access License (CAL) per user for this feature – it’s an Enterprise level feature and you pay for it. What is not so apparent unless you dig around the licensing site is that you also need the Volume Licensing version of Outlook 2010 or 2007 called “Pro Plus”. An OEM or Retail copy of Outlook will not cut it.

Where does this leave them then? Small companies who have shelled out for OEM/Retail copies of Office Professional cannot afford to simply purchase a whole new VLK copy and upgrade. You can’t upgrade and OEM/Retail license to a VLK license, there’s no path. These companies have paid for the Enterprise CALs to use Enterprise features, only to find out that it’s not just the CAL they need!

To me, this is a BIG flaw in the way Microsoft are selling Exchange 2010. Licensing is complex enough without adding this sort of gotcha to a solution, and the companies have paid for an Enterprise CAL. They’re not trying to use an Enterprise feature on a Standard license, they’ve paid for it!

And people like me can’t turn round and recommend an upgrade to a client without upgrading the entire Office licensing too. I hope Microsoft sort this out, I really do, because in all honesty, it puts a real downer on an otherwise superb product that up til now, I have had no hesitation in recommending.

I normally make it a rule that what I post on here is a solution, unfortunately in this case the solution is expensive and involves upgrading your licensing.

While using the New-TestCasConnectivityUser.ps1 script to create a test user for Exchange 2010’s connectivity testing, I ran into an issue:

CreateTestUser : Mailbox could not be created. Verify that OU ( Users ) exists and that password meets complexity requirements.
At C:\Program Files\Microsoft\Exchange Server\V14\Scripts\new-TestCasConnectivityUser.ps1:255 char:27
+   $result = CreateTestUser <<<<  $exchangeServer $mailboxServer $securePassword $OrganizationalUnit $UMDialPlan $UMExtension $Prompt
+ CategoryInfo          : NotSpecified: (:) [Write-Error], WriteErrorException
+ FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,CreateTestUser

Oddly enough, that OU does exist (as it will by default on any Windows Domain!) and the password complexity more than satisfied the complexity requirements. The issue is simple enough to fix, I opened the script in notepad and found the line beginning “new-mailbox” – and deleted the parameter “–OrgainisationalUnit:$OrganistationalUnit”. This means the new user defaults to the default OU – Users!

Just a simple fix to save some time! Thanks MS for the buggy script!

Update: Looks like this occurs when there's more than one OU called Users - my fix will still sort it, but at least you know!

Disclaimer: this post is more for my own recollection than anything else! When it comes to Linux, I’m an amateur and everything I do from the simplest thing upwards is copy-and-paste from much more informed bloggers and websites!

My home server is running Ubuntu Linux 10.10 – access is via an SSH client only. I run an NFS file server for my home network, which stores my Music and Video for the network, and is running an iTunes server. Most of my DVDs have been ripped to high quality MP4 files for viewing on PC, but they aren’t suitable for my iPhone, so I also frequently compress them for viewing on that device.

The following command lists the files in the Source folder and runs Handbrake via the CLI to convert the files using the iPhone template and spits them out to the Video folder.

for file in `ls /media/Data/NFS/Source/`; do $(HandBrakeCLI -v -i /media/Data/NFS/Source/${file} -o /media/Data/NFS/Video/"${file%.vob}.mp4" --preset="iPhone & iPod Touch"); done

For some reason this time the installation of ESXi didn’t pick up the additional NIC and, after seating and reseating the card, checking the UEFI (new-fangled BIOS) settings and getting to the point of ordering a return code from our suppliers, I thought I’d try installing the drivers… you know, just in case.

What you need:

1. VMware vSphere CLI installed on your admin machine (mine’s a Windows 7 Desktop)
2. The latest driver CD for your ESXi component and burn to a CD or mount the ISO
3. Shutdown any running VMs and set the host to Maintenance mode

From there, open a CLI window, which looks just like a command prompt, because it is. Run the vihostupdate.pl script:

vihostupdate.pl --server  --username  --password  --install --bundle “e:\path\to\bundle.zip”

Reboot the host and the new hardware will be installed and ready to use.

I am pretty sure that I didn’t need to do that on any of the other identical x3650s with Intel NICs that I have set up over the last year, so what has changed? The install media I used was freshly downloaded, so that could be a factor, as could the UEFI version on the server. Whatever it was, it won’t surprise me again!

Given I am unlikely to have any profound revelations I shall simply focus on what I have discovered as useful and helpful on my travels!

First up then...

I have been asked on many occasions by individuals and SMEs should they opt for a brand such as HP over Dell or vice versa... as ever my default response is a brief needs analysis. It seems all to often folk get caught up by hearsay/brand loyalty over suitability.

For example I have had both great and poor experiences with well know brands such as HP and Dell.

More importantly our job in IT is to find out what the need really is! And then source a suitable cost effective solution that will last at least for the duration specified. Of course I am simplifying the matter a great deal but in a nutshell that is what we must do and what really should be expected of us. I have lost count of the mount of times I have come across IT solutions that are utterly over kill or completely inadequate and therefore cost a business large sums of time and money.

In short if you get the needs analysis and scoping right the solution will present itself readily and should deliver exactly what is required for the duration specified!

1. Branding. Whether I like it or not, “McGeown” is not easy to spell or remember so I wanted to move away from it as an identity. After some discussion with my friend and creative advisor Matt Hellyer, I picked DefinIT.
2. I’d like to add some more bloggers to the site – the number of visitors is directly in proportion to the number of posts, and I can’t post as much as I’d like to. I will introduce those bloggers as and when they arrive, but they will be people I know and trust – with real technical expertise. (That said, if you’re reading and are interested in writing for DefinIT, please contact me!)
3. Eventually I’d like to use McGeown.co.uk for a more personal site, aimed at family and friends. That’s what it was originally aimed at, and hopefully one of these days I’ll get the time.

Feel free to let me know what you think about the new theme, new name and new URL in the comments below. This blog will always be dedicated to providing high quality IT help on a broad range of subjects, as ever, I hope it helps!

Sam

This is the script:

$ActiveSyncDevices = @()

ForEach ($Mailbox in Get-Mailbox –Server MBX01) {
Get-ActiveSyncDeviceStatistics -Mailbox $Mailbox.Identity –ErrorAction SilentlyContinue | Select DeviceFriendlyName, Devicetype, DeviceUserAgent | ForEach-Object { $_ | Add-Member –MemberType NoteProperty -Name "MailboxIdentity" -value $Mailbox
$ActiveSyncDevices += $_ }
}

$ActiveSyncDevices | Export-csv c:\Path\To\File.csv

Walking through the script it gets all the mailboxes from the server MBX01, gets an object containing the ActiveSync device name, type and user agent. It then adds a property to that object called “MailboxIdentity” and adds it to that object. That object is then added to an array of objects called ActiveSyncDevices, which is then exported to CSV.

For years I have preached to users about the importance of strong passwords, regular password changes and non-proliferation of the same password, yet I've fallen foul of 2 of my own rules. My password is strong - 13 characters, random alpha-numeric, upper and lower case and including special characters - but has been re-used in a few places, and hasn’t been changed in a (long) while.

I do use different passwords for specific things: my online banking, for example, has a unique password. But in other cases the same password has been used in multiple places – e.g. social networking sites, iTunes account, the DNS management for this domain, my email account. Even as I write this list it’s a bit alarming! It’s laziness, and to some extent an inability to remember a password for the dozens of accounts I have.

One Password to Rule them All

I’ve been opposed to using password keepers such as LastPass, KeePass, RoboForm, et al because they have just one password to unlock them, and then the proverbial horse has bolted. I’ve now come to the conclusion that this is more secure than password re-use – at least if there’s one password to rule them all, I can make it a good one.

I’ve tried a “tiered” password system – one password for social networks, one for email accounts, one for financial accounts, and this does improve on the password re-use situation, but it still means if one is compromised they all fall.

Installing LastPass

Installing LastPass is a simple affair:

Securing your passwords

The simple fact of the matter is that just installing LastPass will not secure your passwords – you can still use the same old ones, just auto-filled by LastPass.

There’s a handy tool on the LastPass website called the LastPass Security Challenge that can help see how secure your passwords are – the image on the right is how secure my passwords were to start. Yes, 10.9% – shocking.

To secure your passwords, you need to create a strong password for each and every account you have. Yes, I know that’s a pain in the… proverbial. If an attacker compromises your Twitter password, you don’t want them accessing your Amazon account and ordering £1000 of death-metal do you? It’s not as paranoid as you might think.

With LastPass installed when you browse to a website it will recognise a login and ask if you want to save it. If it has a login stored for a website you can set the option to auto-login.

It will also recognise when you’re changing your password and update the stored password for each site.

With a bit of fiddling, and a password generator, I have now updated all my saved website passwords to strong, unique passwords. It will take a bit of discipline to maintain them, but I think that LastPass will help!

Password strength

I have randomly generated the passwords from https://secure.pctools.com/guides/password. The settings I used to generate the passwords were as follows:

• 12 characters in total
• Uppercase letters [A-Z]
• Lowercase letters [a-z]
• Numbers [0-9]
• Special characters [!*_£$%^&]
• No similar characters [e.g capital i and lowercase l, zero and capital o]

Shamed websites

The following websites wouldn’t let me use a secure password:

eBuyer.com, London Stock Exchange – no special characters allowed

HP.com – no special characters allowed AND maximum 8 characters

The process involves simply opening the correct ports and protocols between the TMG servers and the SCOM management servers, which after a few attempts watching the live logs, I found.

Creating the SCOM 2007 R2 Agent Push Install Rule

From the Threat Management Gateway Server, Launch the New Access Rule Wizard

Obviously we need to allow protocols

This is where it gets specific – add the following “standard” protocols from the Add Protocols dialogue (the port, protocol and direction are here for reference only, you can’t edit them):

Now add in some custom rules – these can be named what you like. The “NLB (DCOM Dynamic)” rule already existed for the NLB TMG cluster – it allows WMI querying for the SCOM agent install.

The protocol rules should look something like this:

In the Access Rule Sources page of the wizard, add a Network Entity and create a new Computer Set that contains all your SCOM management servers for this site – in my case I called the group “SCOM 2007 R2 Servers”

For the Access Rule Destinations, add “Local Host” from the “Networks” group (this will allow to all servers in the TMG array):

Keep the default user set “All Users”

Create and apply the rule.

Creating a SCOM 2007 R2 Agent Communications Rule

To allow the Agent to communicate back to the management server, start another Access Rule wizard from the TMG console to allow “System Center Operation Manager Agent” and “System Manager Operation Manager Agent Installation” from “Local Host” to your “SCOM 2007 R2 Servers” computer set, applicable to “All Users”.

Create and apply the rule, and you should have something like this:

Pushing the SCOM 2007 R2 Agent to Threat Management Gateway Servers

You should now be able to “Discover” your TMG servers in the normal way from the SCOM console – if you have issues I would highly recommend setting a filter on the IP addresses for your SCOM and TMG servers while you do the discovery and installation of the agent – investigate any blocked connections that you might see.

Don’t forget to install the SCOM 2007 R2 Management Pack for TMG. Another useful page is Troubleshooting Issues When You Use the Discovery Wizard to Install an Agent, as is Agent discovery and push troubleshooting in OpsMgr 2007.

When it comes to servers you really want to audit, those that are by definition more at risk from security breach because they are publicly accessible, it’s not so straightforward. Take for example that web server, or FTP host in your DMZ, certainly not domain joined and probably bombarded by daily brute force password attacks. Select the SCOM agent in the console and enable Audit Collection Services?

Nope.

But as you have SCOM already installed, you’ve either set up a Gateway server, or used Certificates to secure communication between the server and your Management Server – so it should be OK, right?

Nope.

To reduce the possibility of interference between Forwarder and Collector, routing through a Gateway is not allowed. Communication must be direct between the two, which means that even your remote domain featuring a Gateway server must allow ALL servers you wish to collect security data from to communicate with ACS directly.

Configuring the ACS Collector to use Certificates

I’m going to assume that you’ve already configured SCOM to communicate using Certificates and that your ACS Collector is on the same server as your Management Server.

- On the ACS Collection Server, open up a command prompt (as administrator, if you’re Server 2008) and navigate to the %systemroot%\system32\security\adtserver

Stop the ADTServer service using “net stop adtserver”

- Configure the certificate for the ADTServer to use with the command “adtserver –c”. This brings up a list of certificates installed on the server that are suitable for ACS – type the  number of the certificate you want to use.

- Start the ADTServer “net start adtserver”

Configure Dummy Computer Accounts with X.509 certificate mapping

- You need a copy of the SCOM certificate from your Forwarder servers on a domain member server with Active Directory Users and Computers access. If you don’t have a copy of the certificate, you can export it from the Forwarder server itself using the Certificates snap-in in MMC, but make sure you save it in DER encoded X.509 format. I use the machine name to name the certificate file so it’s easy to identify.

- Open Active Directory Users and Computers and create an OU to contain the dummy accounts. I created one called “SCOM ACS Forwarders”

- Right click in the new OU and create a new Computer object with the same NETBIOS name as the Forwarder server you wish to collect. Under the View menu, make sure “Advanced” is selected. Right click the new computer object and select “Name Mappings…”. Under the X.509 Certificates tab, click Add and import the SCOM Certificate you exported earlier.

Configure the ACS Forwarders

On each server that will be forwarding to the Collector, log onto the server and open a command prompt. For Server 2008, run as administrator.

- navigate to "%systemroot%\system32\”

- Import the certificate for the agent by running “ADTAgent.exe –c” and selecting the correct certificate from the list.

- Open your Operations Manager Console, select the Monitoring tab and expand Operations Manager > Agent and select the “Agents by Version” panel.

- Select up to 10 agents at a time, and in the right hand Actions pane you can “Enable Audit Collection”

I figured that the PowerShell extensions for managing IIS would do nicely – they are great for managing IIS sites after all and the idea with FTP in IIS7 is that it doesn’t care which protocol you are using, HTTP, HTTPS, FTP or FTP over SSL – all should be configured the same way.

The security setup had to follow the policy that all IP addresses are denied unless explicitly allowed.

I imported the module and listed the cmdlets:

Import-Module –Name WebAdministration

Get-Command –PSSnapin WebAdministration

Creating a new FTP site

This is very, very simple with PowerShell

New-WebFtpSite -Name 'FTP Site' -IPAddress '192.168.10.22' -HostHeader 'ftp.mcgeown.co.uk' -PhysicalPath 'd:\FTP\

A good start then!

By default, the site will allow any IP address to access the server – it’s time to start locking it down.

Allow Authorised Users

1) Add-WebConfiguration -Filter /system.ftpserver/security/authorization -PSPath 'IIS:\' -Value @{accessType='Allow';users='*';permissions=3} -Location 'FTP Site'

Restricting IP Access

Unfortunately the IP Address Restrictions part of the configuration isn’t exposed directly by a cmdlet so I thought I’d use one or two of the lower level IIS configuration cmdlets – Add-WebConfiguration, and Set-WebConfigurationProperty. And after a lot of fiddling and a lot of help by a colleague, I stumbled upon the correct syntax:

2) Set-WebConfigurationProperty -Filter /system.ftpserver/security/ipsecurity -Name allowUnlisted -Value $false -Location 'FTP Site' -PSPath 'IIS:\'

3) Add-WebConfiguration -Filter /system.ftpserver/security/ipsecurity -PSPath 'IIS:\' -Value @{ipAddress=’192.168.1.1’;subnetMask=’255.255.255.255’;allowed=$true} –Location ‘FTP Site’

The following code is appended to the end of the applicationHost.config file – now hopefully you can see that the crazy colours relate to the section in the same colour on below:

<location path="FTP Site" >
<system.ftpServer>
<security>
<ipSecurity allowUnlisted="false">
<add ipAddress="192.168.1.1" allowed="true" />
</ipSecurity>
<authorization>
<add accessType="Allow" users="*" permissions="Read, Write" />
</authorization>
</security>
</system.ftpServer>
</location>

Scripting the whole thing

I’ve created two files, Add-AllowedIPs.ps1 and AllowedIPs.config. The config file is just an XML structure with the allowed IPs and Site name:

<?xml version="1.0"?>
<Configuration>
<Site name="FTP Site">
<Address>
<IP>192.168.8.1</IP>
<SubnetMask>255.255.255.255</SubnetMask>
</Address>
<Address>
<IP>10.10.10.0</IP>
<SubnetMask>255.255.254.0</SubnetMask>
</Address>
</Site>
</Configuration>

The ps1 file is a simple script to read the configuration and apply the settings:

$ConfigFile= "AllowedIPs.config"
if(Test-Path $ConfigFile){\[xml\]$Config = Get-Content $ConfigFile}else{throw ("Unable to find configuration file: " + $ConfigFile)}

$Site = $Config.Configuration.Site.Name
Write-Host "Clearing $Site IP Security Lists" -BackgroundColor Yellow -ForegroundColor DarkBlue
Clear-WebConfiguration -Filter /system.ftpserver/security/ipsecurity -PSPath 'IIS:\' -Location "$Site"
Write-Host "Adding Allowed IPs to $Site" -BackgroundColor Yellow -ForegroundColor DarkBlue
$Config.Configuration.Site.Address | foreach {
$IP = $_.IP.Trim()
$SubnetMask = $_.SubnetMask.Trim()
Add-WebConfiguration -Filter /system.ftpserver/security/ipsecurity -PSPath 'IIS:\' -Location "$Site" -Value @{ipAddress="$IP";subnetMask="$SubnetMask";allowed=$true}
}

Not quite sure why doing something so simple should have occupied so much of my time, but I hope it saves you some!

Sam

It’s good practice to separate VMotion, virtual machine and iSCSI traffic, it also helps you manage those logical and physical connections.

Connect to your ESXi server using the vSphere Client and select the host. Go to the configuration tab and click “Add Networking…”. Select a new VMKernel connection type.

Create a new virtual switch, but at this point don’t assign any network adaptors.

Create a new Port Group. If you are planning to use Jumbo Frames, you can’t use this Port Group – to specify a larger MTU you have to create the Port Group using the command line interface. If you aren’t using Jumbo Frames for whatever reason, you can go with the standard wizard-created Port Group.

Set the IP address for the VMKernel – this address needs to be on the same network as the iSCSI SAN because the iSCSI protocol does not like to be routed.

Click through the summary page and your new vSwitch and VMkernel is created

If you’re using Jumbo Frames, you need to delete the newly created Port Group, so open the vSwitch properties and Remove it

You should now have an empty vSwitch

Open up your command line interface – this is either a vMA instance, or if you have enabled SSH on the ESXi server, then the SSH terminal. Log in as a user with permission to sudo. I am using SSH.

Just like on a physical switch, the vSwitch needs to be told to allow Jumbo Frames by setting the MTU to 9000.

esxcfg-vswitch --MTU=9000 vSwitch4

Next we can add the Port Groups. iSCSI multipathing requires one for each pNIC we are planning to utilise. To add the new Port Groups, use the following commands:

esxcfg-vswitch --add-pg=iSCSI1 vSwitch4
esxcfg-vswitch --add-pg=iSCSI2 vSwitch4

If you need to specify the VLAN for the iSCSI traffic, use the following commands:

esxcfg-vswitch --vlan=105 --pg=iSCSI1
esxcfg-vswitch --vlan=105 --pg=iSCSI2

Now we need to add a VMKernel IP address per pNIC we are using to the Port Group on the iSCSI network. Make sure you use the next number in the vmk sequence, e.g. if you already have vmk0, vmk1 and vmk2 configured, create the next one as vmk3. To see your currently configured VMKernel ports, use esxcfg-vmknic –l.

esxcfg-vmknic --add vmk3 --ip 10.20.200.100 --netmask 255.255.255.0 ↓
--portgroup iSCSI1 --mtu=9000
esxcfg-vmknic --add vmk4 --ip 10.20.200.101 --netmask 255.255.255.0 ↓
--portgroup iSCSI2 --mtu=9000

If you switch to the vSphere Client and refresh, you should see the new config

Now we need to add the physical adaptors to the virtual switch. What we don’t want to do is add two pNICs to the iSCSI kernel, it doesn’t like this one bit! Open the properties for the vSwitch and go to the Network Adaptors tab. Click add, and select the pNICs you are going to use, and, this is important, set one pNIC to Active and the rest to Standby.

Now go to the iSCSI1 Port Group under the Ports tab and Edit the configuration – override the Failover order and set one pNIC to active, and the other to Unused. This gives you a one-to-one mapping from the Port Group to the pNIC. Edit the second iSCSI Port Group and map the second pNIC to active and the first to unused.

Obviously if you’re using more network adaptors you have more port groups and map them one-to-one. Switch back to the SSH session and bind the VMKernels to the Software HBA. You need to check the name of the Software iSCSI HBA from the Storage Adaptors

Add the vmk ports you created earlier

esxcli swiscsi nic add -n vmk3 -d vmhba34
esxcli swiscsi nic add -n vmk4 -d vmhba34

Add your iSCSI targets in the normal way, rescan the software adaptor to pick up all the paths. Modify each LUN to use Round Robin path selection.

You can verify both paths are being used using ESXTOP to in the SSH session and pressing “n” for network.

References:

iSCSI SAN Configuration Guide - http://www.vmware.com/pdf/vsphere4/r40/vsp_40_iscsi_san_cfg.pdf

Troubleshooting Software iSCSI on ESXi - http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1008083

Both TMG servers will be domain joined – there are several reasons for this, not least of which is that we require integrated authentication for the proxy clients.

The array will be a Network Load Balanced proxy server for all LAN clients to access the internet; it will provide content caching and malware protection. It will also reverse proxy Outlook Web Access, ActiveSync, Outlook Anywhere, SharePoint and some internal static HTTP resources. It will also provide SSL VPN (SSTP) access for remote clients, but that will be the subject of a later post.

The network diagram for this setup looks like this:

NIC Configuration

Prior to installation, it’s important to get the network adaptors configured correctly, the reasons for which are discussed later in more detail, but the basic rule is that there should only be one default gateway and one adaptor configured to provide DNS resolution to the internal network.

TMG01

	Internal	Perimeter/DMZ	Intra-array
IP Address	192.168.8.28	192.168.10.9	10.20.200.1
Subnet Mask	255.255.255.0	255.255.255.0	255.255.255.248
Gateway	[none]	192.168.10.1	[none]
DNS 1	192.168.8.22	[none]	[none]
DNS 2	192.168.8.5	[none]	[none]

TMG02

	Internal	Perimeter/DMZ	Intra-array
IP Address	192.168.8.29	192.168.10.10	10.20.200.2
Subnet Mask	255.255.255.0	255.255.255.0	255.255.255.248
Gateway	[none]	192.168.10.1	[none]
DNS 1	192.168.8.22	[none]	[none]
DNS 2	192.168.8.5	[none]	[none]

Installing TMG01 and TMG02

Run the following procedure on BOTH servers.

Configure all 3 NICs with their assigned IP addresses.

Run the TMG 2010 installer, ensure you’ve run Windows Update and you’re fully patched.

Run the Preparation Tool, which basically installs pre-requisites and checks for required Roles and Features. Correct anything that’s not installed.

Launch the Forefront Configuration Wizard

Accept the license agreement

Enter the product license details

Select the option to install

Install path (this is not where the proxy cache is stored, configure that later!)

Configure the Internal network IP address ranges.

Accept the services warning

Since there are domain controllers in other network ranges for this domain, TMG will create a system policy for AD connections to all the IP addresses of Domain Controllers. It will also create a management access rule for the workstation IP that is connected

Click Install to begin the installation – this takes a while!

After about 40 minutes, you reach the completion screen.

Configure Internal DNS records

While that’s installing, you can set up the required DNS records.

TMG01*	192.168.8.28
TMG02*	192.168.8.29
proxy (or whatever you want to call the proxy NLB!)	192.168.8.30
TMG01-IA	10.20.200.1
TMG02-IA	10.20.200.2

*created automatically as they’re domain joined with AD integrated DNS

The additional records here are for the NLB virtual IP address (proxy.domain.com) and for the Intra-array communications network. The intra-array DNS records are required because of the Kerberos dependency to access the Array Configuration Storage Server (CSS); it requires a SPN (Service Principle Name) to be configured, which in turn requires DNS resolution.

Update TMG to latest SP level (SP1 at time of writing)

Both TMG servers should be updated to the latest SP level.

The service pack installer must be run from an elevated command prompt, otherwise it will fail.

Run the SP installation

Accept the license agreement

Select the server, and the credentials

Run the installation

If you receive any errors or warnings, the chances are you’ve not run the installation from an elevated command prompt. You need to re-run the SP to install correctly.

Restart the server (it now seems to take a lot longer to reboot the servers – be patient!)

Run the TMG configuration Wizard

When you run the Forefront Threat Management Console for the first time it will run the Getting Started Wizard

Click, Configure network settings – start the network setup wizard

Select the correct network template – for this setup it’s Back Firewall

Select the network adaptor for the LAN network – this is where some network routing information is required. TMG should not have multiple default gateways, just one “gateway of last resort”. In this case it should be the address of the Front firewall on the DMZ, because that’s where we want to route anything that is not on our internal network. This means that we need to add a route for all INTERNAL networks that we want to route via the gateway for the internal network. This depends on the topology of the network TMG is installed in. DNS is configured on the interface closest to your internal DNS that is capable of resolving external namespaces. Routing is configured in the next section.

Select the network adaptor for the perimeter network, and set the type of IP addresses to use (in this setup Private as there is the Front firewall between the External network and the DMZ: there will be Network Address Translation between the two and we need to avoid “double-NATing” ).

Finish the wizard and run the “configure system settings” wizard As the machine is already named and part of the Domain, there’s no need to change any of the current settings – click through to finish

Now run the “define deployment options link and set the Microsoft Update, then Network Inspection and Web Protection settings, CEIP settings, and feedback. I kept the defaults here, they can be changed later if required.

The wizard is now complete, aside from the web access wizard which when you click close.

Configure the Web Access Policy Wizard

Create the default web access URL categories

Add any non-default URLs you wish to block

Configure any unrestricted users, if required.

Enable/disable Malware Inspection, depending on your license.

Configure HTTPS inspection – we will allow users to establish HTTPS regardless. You can configure to block invalid HTTPS certificates.

Configure the Web Cache – best practice is to use a non-system drive for caching

Review and finish the setup

Check “Save the changes and restart the services”

Routing Configuration

With the additional subnets in the network, the TMG servers need to know where to route traffic, which interface they are to route the traffic on, and how to treat the network. The routing is easiest to configure at the server level using the ROUTE command.

View the routing table using the “route print” command from an elevated command prompt.

You can see the default route, or gateway of last resort is set to 192.168.10.1 – the DMZ gateway. You can also see the server has a route for the networks it has an IP address on; namely the LAN and Intra-array networks (192.168.8.0/24 and 10.20.200.0/29). These are “On-Link”, meaning anything for those networks will be sent via the NIC with the IP address on the same subnet.

The problem with this configuration as it stands is that the server can only route internal networks that it has an IP address on, or send it to the default gateway on the DMZ. When you try and access an internal network (e.g. 10.20.110.0, which for me is the the client network), the only route it has is the default gateway in the DMZ which does not have access to the internal LAN.

To correct this we add a route for each internal network, specifying the correct gateway and interface.

route add –p [network IP] MASK [subnet mask] [gateway IP] METRIC [metric] IF [Interface]

“Route add” is self-explanatory, “–p” sets the route as persistent across reboots. For the client network above, [network IP] is 10.20.110.0 and the [subnet mask] is 255.255.254.0. The gateway needs to be the gateway for the network that the LAN NIC is on – 192.168.8.1. The metric is the priority if the gateway, lower numbers are a higher priority – I am using 20. To find the [interface] the easiest way to be sure is to check the network adaptors properties and make a note of the MAC address, and then compare it to the Interface List in the route show command. The LAN NIC is interface 11 in this configuration. The final route command looks like this:

route add –p 10.20.110.0 mask 255.255.254.0 192.168.8.1 metric 20 if 11

You can now test the route by pinging a host on the new subnet. Add routes for all internal networks you want routed through the LAN interface.

Configure Service Principle Names for Intra-array communications

CSS access over the Intra-array network requires Kerberos, which requires SPNs. To create the SPNs, run a command prompt and issue these commands (per server):

setspn -a ldap/[Intra-array DNS Name] [NETBIOS Name]

setspn -a ldap/[Intra-array DNS Name] :2171 [NETBIOS Name]

For example:

setspn –a ldap/TMG01-IA.DOMAIN.COM TMG01

setspn –a ldap/TMG01-IA.DOMAIN.COM:2171 TMG01

To check the SPNs are correctly configured, use “setspn –L [NETBIOS Name]”

This needs to be configured on each array member.

Preparing and configuring the Standalone Array

On TMG01 - In the Threat Management Gateway console, under Firewall Policy > Toolbox (on the right hand pane), expand Computer Sets and add TMG01 and TMG02 to the groups “Managed Server Computers” and “Remote Management Computers”

On TMG02 – open the Threat Management Console, select the server and select “Join Array” from the Tasks tab

Select the option to join a stand-alone array

Enter the IP address for TMG01 on the intra-array network as the array manager, since this is a domain environment use the credentials of the logged on user

Finish the wizard

The wizard will then copy the configurations from the designated Array Manager, this takes short while and will probably disconnect your session if managing remotely.

Back on TMG01, open the System pane and look at the Servers tab – hit refresh if you can’t see the second server. You should see TMG01 and TMG02, with TMG01 reporting as the Array Manager and TMG02 reporting as the Array Managed. It may take a while for the egg timer to turn to a green tick for the managed server.

Right click each server in turn and select properties, then select the Communication tab. Under Intra-Array Communication and select the IP address on the intra-array network.

From the Roles Configuration page, open the “Configure Array Properties” and change the DNS Name to the intra-array DNS name configured earlier

It may take a while, but the array will sort itself out

You can also check the synchronisation status under the monitoring tab

Creating a rule Enabling NLB communication on Intra-array Network

Under the Firewall Policy tab, go to the Toolbox and expand the Protocols pane. Launch the New Protocol Definition Wizard

Create new protocol as follows (some places recommend port 10000-11000 but the firewall logs show connection attempts up to port 14000)

Click through the rest of the wizard, keeping defaults and apply the configuration change

Switch to the Tasks tab and click “Create Access Rule”

The rule is to allow traffic

On the new customised Protocol

From Local host and the Intra-Array Communications network

To the same destinations

Apply the new rule to all users, finish the wizard and apply to the server – wait for the array to converge again before moving on

Enabling Network Load Balancing

From the networking tab, click the “Enable Network Load Balancing Integration”

Enable load balancing for the internal and perimeter networks

For each network configure the Virtual IP address and Cluster mode by clicking “Configure NLB Settings…”

The Primary VIP for the internal network is the IP we registered earlier for proxy.mdlimited.com (192.168.8.30). The external VIP is the next IP address in the DMZ. If the Cluster Operation Mode is set to Unicast mode the vSwitch on the server must be configured as per the VMware recommendations for NLB configuration – it must accept forged MAC addresses and must not send updates to the physical switch

Sources: Microsoft NLB not working properly in Unicast Mode, Sample Configuration - Network Load Balancing (NLB) UNICAST Mode Configuration

It takes a few minutes for the configurations to converge and for the NLB to come online, check the Configuration tab under Monitoring, and look at the Services tab to see that the NLB service is running in both servers

At this point the forward proxy NLB should be operational so you can change your browser to point to proxy.domain.com and test it.

To finish up the external NLB setup, you simply map the public IP address on your front firewall to the DMZ NLB IP address. You can then use any of the publishing wizards to publish OWA, HTTP or other internal services. If you need additional IP addresses, remember to use the NLB properties in the TMG console to add a new VIP rather than the NLB MMC.

Since this post has well exceeded 2000 words, I’m going to leave it there – I hope this has been informative, please leave feedback!

As per this article on virtualkenneth.com, you need to edit the VMX file to change the SCSI card and OS type, otherwise you’ll have a kernel panic on boot.

Once you’ve done that boot the vMA in workstation and check the console, you will be prompted to provide a secure password for the vi-admin user. Once you’ve done that, the network config wizard will start. You can use DHCP to configure the network, then provide a hostname, or you can specify your network settings manually.

The wizard will then reset the network interfaces and bring up the new settings.

To re-run the network config wizard, log onto the console and use the following commands:

cd /opt/vmware/vma/bin

sudo ./wmware-vma-netconf.pl

After this, I was able to access my vMA by SSH – PuTTY is my client of choice.

Joining the vMA to the domain for AD authentication

If you want to use AD authentication with your ESXi hosts, you can join them to the domain – it’s the same with the vMA. To join the vMA use the following command:

sudo domainjoin-cli join <domain-name> <domain-admin-user>

It will then prompt you for the root password (for the sudo) before it prompts for your AD admin password! I had errors with it saying ports weren’t open. The most likely cause for that is time so make sure that your server is running in UTC mode (this matches your ESXi and is compatible with your Domain Controller). These steps are taken from Simon Long at The Slog:

sudo rm /etc/localtime

sudo ln -s /usr/share/zoneinfo/UTC /etc/localtime

If you use NTP to sync your environments time, it might be worth adding in your NTP servers to the vMA.

sudo nano /etc/ntp.conf

Add in your ntp servers under the heading: # Use public servers from the pool.ntp.org project.

Configure ntpd to start on reboot:

sudo /sbin/chkconfig ntpd on

Restart ntpd:

udo /sbin/service ntpd restart

Make sure your NTP servers are reachable:

sudo ntpq -p

The final domain join looks like this (sorry for redaction, one day I will have my own test environment!):

Some ease-of-use tips over at VirtuallyGhetto.com allow you to set a default domain for login, and groups who can sudo.

You will need to edit /etc/likewise/lsassd.conf and uncomment "assume-default-domain = yes" and then save your changes:

sudo vi /etc/likewise/lsassd.conf

You will need to reload the configurations for the changes to take effect by running the following utility:

sudo /opt/likewise/bin/lw-refresh-configuration

Now, you can login by just specifying the username without having to provide the full AD domain name.

Just being able to log in is not much help – the power is being part of the sudoers group. Edit /etc/sudoers using vi-admin account, make sure you use 'sudo':

sudo vi /etc/sudoers

Add the following towards the bottom of the file:

%DOMAIN\\VMware\ Admins ALL=(ALL) ALL

Note: We're escaping both the initial back slash and the space – the group is “DOMAIN\VMware Admins”

I’m now able to log in as a domain user, and SUDO:

Adding an ESXi Server to the vMA

To add an ESXi server to vMA is dead simple, especially if your ESXi server is using AD integration too.

sudo vifp addserver <server FQDN> --authpolicy adauth --username DOMAIN\\user

You can verify the managed servers using “sudo vifp listservers –l”

Test the connection, and list the NICs

Success!

Conclusion

The vMA Config guide is great for getting to know and configure the vMA – as is the vma-help command. I’ve linked to articles that helped me along the way, but it’s worth going through the rest of Simon Long’s article and configuring the vMA as a syslog server for your ESXi hosts – they lose their logs on reboot!

To create a new Mailbox Database for the Personal Archives to be stored in:

New-MailboxDatabase –Name “Archive Mailbox Store” –EdbFilePath “D:\Path\Archive.edb”

To enable a specific user’s archive [and specify the database]:

Enable-Mailbox “User.Name” –Archive [–ArchiveDatabase “Archive Mailbox Store”]

To enable ALL existing user’s archives:

Get-Mailbox | Enable-Mailbox –Archive [–ArchiveDatabase “Archive Mailbox Store”]

To move the user’s Archive to the Archive Mailbox Store:

New-MoveRequest “User.Name” –ArchiveOnly –ArchiveTargetDatabase “Archive Mailbox Store”

PowerShell is the way forward, so it’s best to learn the command for the EMC (Exchange Management Shell). If you simply HAVE to use the GUI, here it is:

Enable Personal Archive:

Move Personal Archive to Archive Database

References

The starting point is always working out if you *need* to upgrade – what’s the business argument. For that you need to look at what’s new in Exchange 2010 SP1, the release notes and prerequisites. Finally, the installation instructions for upgrading from Exchange 2010 RTM to SP1.

It’s important to note, as mentioned in the upgrade doc:

Note: After you upgrade to Exchange 2010 SP1, you can't uninstall the service pack to revert to Exchange 2010 RTM. If you uninstall Exchange 2010 SP1, you will remove Exchange from the server.

High level plan

Back up Active Directory and Mailbox Stores

Install the Exchange 2010 SP1 prerequisites

Upgrade the Edge Transport server

Upgrade the Hub/CAS/Mailbox server

Installing the prerequisites

All the advice I’ve seen so far is that the installer does a bad job at detecting and installing the prerequisites for SP1, with complaints about it leaving the server in an inconsistent state, and incorrect links, so I will install the prerequisites manually. There is an easy matrix to work out what hotfixes are required on the Exchange team blog -  http://msexchangeteam.com/archive/2010/09/01/456094.aspx.

Both my servers are running Windows Server 2008 R2, so the hotfixes requires are

http://go.microsoft.com/fwlink/?linkid=3052&kbid=979744

http://go.microsoft.com/fwlink/?linkid=3052&kbid=983440

http://go.microsoft.com/fwlink/?linkid=3052&kbid=979099

http://go.microsoft.com/fwlink/?linkid=3052&kbid=982867

Some of these may well have been installed on the server already, but I downloaded them all to the same location as the SP1 file too so that the resources were available should I need them. Then I just went through the list installing, first on the Edge, then the Hub/CAS/Mailbox. Each hotfix does request a restart, but you can install them all and restart once.

Now is also a good point to install the MSMQ Server component on your CAS server, and also the Microsoft Office 2010 Filter Pack – you’ll see why when you get to my experiences installing the update to my Hub/CAS/Mailbox server.

Upgrading the Exchange 2010 Edge Transport Server

There is a little bit of confusion in the docs around the order in which to install the roles – the clearest explanation in the Exchange blog says:

The Edge Transport server role can be upgraded at any time; however, we recommend upgrading Edge Transport either before all other server roles have been upgraded or after all other server roles have been upgraded

You can do it either way you like, I am installing the Edge first simply because I can see the process before upgrading the Hub/CAS/Mailbox server.

Running as a local administrator, I fired up the installer, which prompts and extracts the installer to a specified location

Run the Setup.exe – annoyingly, you have to select the language options first (why can’t it update the languages INSTALLED on the server – it’s an unnecessary step!)

Introduction

License agreement

If you got all the prerequisites right before, you should have a green light across the board (if not, I suggest you manually install them as discussed earlier).

And if all goes well, you’ll get the confirmation

I didn’t need to restart after the installation, I tested mail flow in and out of the server just to make sure everything was OK – it was.

Upgrading the Hub/CAS/Mailbox server

Having used the Edge server as a bit of a test installation, running through the installer for Hub/CAS/Mailbox was just that, running through.

The prerequisites did fail on this hotfix (http://go.microsoft.com/fwlink/?linkid=3052&kbid=977020) and the Microsoft Office 2010 Filter Packs, both of which annoyingly did require a restart.

Once all prerequisites were fulfilled, the upgrade runs…and ran happily for 51 minutes, until I recieved this:

The error text is not particularly helpful.

$error.Clear();

Start-SetupProcess -Name "iisreset" -Args "/noforce /timeout:120

Searching through the System Event log, I found this error, pointing to the need for Message Queuing to be installed. This is a pretty fundamental miss by the installer team, not to check for a required service.

Once the Microsoft Message Queuing Server component is installed (and another restart later), I re-ran the upgrade, which thankfully restarts almost where it stopped off. Finally, another 38 minutes later – success!

Bonus section – upgrading Management tools

I thought after all that my upgrade saga was over – not quite! While upgrading the management tools on my local Windows 7 box, I hit another mysterious error:

A Google search based on this error was not helpful at all, the one or two relevant posts were to do with the execution policy for PowerShell. Since I develop PowerShell scripts on this machine the policy is already set to “unrestricted” (Set-ExecutionPolicy unrestricted -  Get-ExecutionPolicy to check). I checked that I had the prerequisites and tried again – only to fail with the same error.

Digging through the setup log, I found references to “tmlisten”

[09/10/2010 09:31:54.0020] [2] [ERROR] Unexpected Error

[09/10/2010 09:31:54.0020] [2] [ERROR] Service 'tmlisten' failed to stop due to error:'Cannot stop tmlisten service on computer '.'.'.

Tmlisten is the Trend Micro service, which requires a password to stop it – once the service was manually stopped, it installed ok. Can’t blame that one on the installer team!

So, I’m going to assume that a) you’re installing the update for a reason, like one of the bugs it fixes and b) you have taken a backup of your OpsManager databases.

I am installing on my x64 Windows Server 2008 with RMS and ACS installed, and a dedicated SQL server box. I have several gateway servers, and quite a few manually installed Agents. The high-level upgrade process looks something like this:

1) Install on Root Management Server (and Audit Collection Services, same server)

2) Update SQL procedures

3) Import the updated Management Pack

4) Update Gateway servers (I have no secondary Management Servers)

5) Update Discovered Agents

6) Update Consoles

7) Update Manual Agents

I’m not going to duplicate the Kevin Holman article, which does a nice job of stepping you through the update – I will however, add my experiences from each step.

1) When you install this on Server 2008 R2, you *MUST* run the installer as a privileged and elevated user. The simplest way to do this is to run a PowerShell prompt as administrator, then kick of the MSI. Simple! Run the Server update first, then the ACS.

Step 2) and 3) completed perfectly well.

Step 4) Again required running the patch installer from an elevated PowerShell prompt, then running the Gateway update. As mentioned in the Kevin Holman article, the patch was not added to the Agent folder, so I manually copied that.

Step 5) Updating the Discovered Agents is just a case of going to the Pending Management view under Administration in the Console and approving the Agents

Step 6) was a simple case of running the installer on the Console machines, 7) we rolled out the patch via SCCM.

First, you need to install the MSI on both ends – for the sake of this, say SERVER1 and SERVER2. Once you’ve installed it on the server, navigate to \Program Files\Microsoft Corporation\NT Testing TCP Tool\, you should see a few copies of the tool for different architectures. Locate the correct .exe for you architecture and rename it to ntttcps.exe. Copy this and rename it to ntttcpr.exe – these form the sender and receiver parts of the tool.

SERVER1 is going to be my sender, and SERVER2 the receiver. On SERVER2, run a command prompt and navigate to the NT Testing TCP Tool folder. A typical listener is kicked of with this command:

ntttcpr –m 1,0,10.1.1.2 –a 6 –fr

Broken down, this command does the following:

-m [Sets a mapping] 1 [with 1 thread], 0 [to CPU0], 10.1.1.2 [listening on IP 10.1.1.2] –a 6 [expect asynchronous data, use 6 overlapped buffers] –fr [always use full receive buffers]

The implicit options are 64KB buffer size, 20,000 buffers on port 5001. This is a 1GB test transfer, so it could take a while over a slow connection. If you want to set the buffer, use –l [size]. If you want to set the number of buffers, use –n [number]. To set the port to use, use –p [port].

Now that the listener is waiting for a connection we can kick off the test from the sender – SERVER1.

ntttcps –m 1,0,10.1.1.2 –a 2

Similar to the previous command on SERVER2, this does the following:

-m [sets a mapping] 1 [with 1 thread], 0 [to CPU0], 10.1.1.2 [sending to IP 10.1.1.2] –a 2 [send asynchronous data, use 2 overlapped buffers]

The implicit options are the same as before. A useful option is to use the –f [filename.txt] to send the output to a text file.

Here’s the results of a test I conducted earlier:

Since Edge, FPE and TMG can now all exist on a single 64-bit server, I will start with a clean installation of Windows Server 2008 R2, up to date with all the latest hot fixes. The server itself is nothing too spectacular, for testing purposes it has 2 virtual CPUs and 2GB RAM. It does need 2 NICs, one on the internal LAN and one on the DMZ. Since the DMZ is behind a hardware firewall, an external IP address has been mapped to the servers DMZ NIC. The server is named EDGE01.

To begin with, I need to install the Active Directory Lightweight Directory Services role, which includes the .NET 3.5 framework. So to start, you can fire up the Add Roles Wizard and install AD LDS, but the simpler way is to use the PowerShell ServerManager commands.

I’m installing the Exchange 2010 prerequisites as per this Technet article: http://technet.microsoft.com/en-us/library/bb691354.aspx.

From an elevated PowerShell prompt, import the server admin module:

Import-Module ServerManager

Add-WindowsFeature NET-Framework,RSAT-ADDS,ADLDS –Restart

The components will install, and the server will restart (leave of the –Restart to restart later).

Now we can move on to the Exchange 2010 Edge role installation. First of all, check that you can resolve the existing Exchange 2010 and your Global Catalog servers, either by ping or NSLOOKUP. Since this server is in the DMZ and LAN, and not a domain joined, I have added the DNS suffix to the Internal LAN NIC. If DNS is not applicable (i.e. only in DMZ), create an entry in the server’s HOSTS file. You also need to create an A record in the internal DNS for the Edge server so that the internal Exchange installation can get to the Edge server.

Run the Exchange 2010 setup wizard, click through the Introduction, read and agree to the License Agreement, check your Error Reporting settings and come to the Installation Type. Select Custom Exchange Server Installation.

Select the Edge Transport Role (Management Tools will select automatically)

Set your CEIP settings to your preference and then the Readiness Checks will begin. Once completed, check that the prerequisites are installed and configured – if not go back and install/configure them and retry the checks until they are all passed.

Review the setup completion window, ensure there are no errors:

This will then open up the Exchange Management Console (EMC). Select the Edge Transport server and have a look through the settings. Now is also a good time to enter your license key, if you’re not using a trial version. You can do this again via the EMC, or the Exchange Management Shell (EMS)

Set-ExchangeServer -Identity 'EDGE01' -ProductKey 'XXXX-XXXX-XXXX-XXXX-XXXX’

If your existing Exchange has been updated (i.e. mine is on Update Rollup 4) then you need to match the update version on the Edge server.

To configure the Edge Server for EdgeSync, use the following command to create an EdgeSync configuration file on your new Edge server:

New-EdgeSubscription –FileName c:\EDGE01-Config.xml

You get the above message, which essentially warns you that a) EdgeSync will overwrite your existing configurations, b) you must now manage the server from your Internal Exchange organisation, c) the Edge and Hub Transport servers must be able to resolve each other’s FQDN, d) you have 1440 minutes (24 hours) to complete the subscription or it will eat your mum and/or dog*. (*may not be true)

If you view the created subscription file, you can see that there’s some pretty obvious configuration data such as the server name and FQDN, a certificate, a user name and password to create, the AD LDS (or ADAM in old money) port and some product version and ID info.

Copy this subscription file over to your Hub Transport server and fire up a new EMS window.

New-EdgeSubscription -FileData ([byte[]]$(Get-Content -Path "C:\EDGE01-Config.xml" -Encoding Byte -ReadCount 0)) -Site "Default-First-Site"

Start-EdgeSynchronization

Essentially this command reads the XML config file and allows you to add the site option – this is required when running on the Hub Transport server. (See http://technet.microsoft.com/en-us/library/bb123800.aspx). Start-EdgeSynchronization does exactly what it says on the tin. If all is successful you should see output like in the screenshot below:

At this point I would suggest testing your mail flow, I sent a few emails to my gmail account and by looking at the headers realised that I needed to add in the SmartHost for my send connector. This is done on the Hub Transport server, under send connectors you should have two EdgeSync Send connectors:

As I have multiple sites and versions, there was some playing with the send connector costs in order to get the mail flowing correctly – but that’s specific to my setup, so I won’t go into it.

As this is now getting to be a long old post, I’m going to split it into two parts – I will link the next one here. In this post I've shown you how to set up the Edge Transport server and get it talking to your Hub Transport – next I’ll show you the FPE install, configuration and also TMG install and configuration. Until then…enjoy!

If you’re having trouble accessing OWA after updating Exchange 2010 with any of the Rollup packages, try this:

- Uninstall the update package from the Programs and Features control panel

- Download the package file directly from Microsoft, don’t use Windows Update

- Open a command prompt or PowerShell prompt as Administrator

- Navigate to the location of the package (.msp) and run from the elevated command prompt.

Apparently when Windows update installs the package it doesn’t run it with the elevated privileges to write to the folder in the Exchange program files – why, I have no idea!

Overall, there’s quite a lot of overlap between the two exams, with the more heavily theory and design based PRO exam being a “high-level” of the more hands-on management and cmdlet based TS exam.

Study materials I used were Technet, SAMS Microsoft Exchange Server 2010 Unleashed and a test installation. I used practice exams from MeasureUp and also spent time answering peoples 2010 based questions on Experts Exchange and Technet Forums.

The problem is, I seemed to be spending more time fixing the blog than writing on it. I’ve had all sorts of problems, ranging from incompatibility with my hosting provider, theme compatibility issues, random code issues and more. Email notifications seem to work, then stop, then work again. Simple things like adding reCAPTCHA support to cut down the enormous amount of comment spam have taken days of head scratching. Whether or not these issues are down to my ignorance or the software, the outcome is the same, I don’t want to be fixing my blogging software, I want it to just work!

Then there’s the search engines, and the existing web presence that I have. A major consideration for me when changing the software is the fact that a lot of my traffic comes from links that are embedded in forums and other blogs – and that search engines respond with the existing BlogEngine posts rather than the newer ones. I’ve considered this, and I think it’s worth the risk. I will leave the BE running for a while and block search engine traffic to it so that direct links in will still be valid. I’ll see where the traffic takes me – but the advantages of changing now outweigh the risks. I will look to do some sort of URL redirect if it becomes an issue.

So  why move to WordPress then? It’s not based on Microsoft technology and it could potentially set me back to square one with my web presence. Quite simply, it just works. It has a massive ecosystem built up around it of plug-ins, themes and widgets. It’s mature – very mature – software that is actively developed and much more widely used than BlogEngine.Net is.

The import of the BlogML from BlogEngine was pretty pain free -  the categories came in as a GUID rather than the friendly name, but that was a simple matter of updating the MySQL table using a query. I’ve decided to slim down the categories, and as such I’ve moved the existing post categories into tags (handy little plug-in that). The theme I am using is nice enough, maybe when I have some time I’ll customise it a bit further.

But, I’ve made the jump; Windows Live Writer is plugged in to WordPress and I am hoping that it all comes together nicely. Recently I’ve been studying for my MCITP: Enterprise Exchange Administrator exams which I’m taking on Monday (70-662 and 70-663), so hopefully I can push some more Exchange stuff this way.

Until then, thanks for reading!

Sam

The environment is a single AD domain with 4 sites, Site1, Site2, Site3 and Site4. In Site1, Site2 and Site3 there are 3 Exchange 2003 servers, one per site. In Site4 there is an Exchange 2007 SP2 server (CAS, Mailbox, HT). All the connectors required worked as expected, and inter-site routing works as expected.

I introduced into the mix a 2010 Enterprise server (CAS, Mailbox, HT) to Site1 as a prelude to a full upgrade of the site to Exchange 2010. When a test mailbox from Exchange 2010 attempts to send to a mailbox in Site1 Exchange 2003, it routes via the Site4 Exchange 2007.

Find out which site (if any) is a transport hub:

Get-ADSite

Find out the Site Link costs, and the Exchange specific costs (if set). Exchange costs override AD site costs:

Get-ADSiteLink

List the Routing Group Connectors:

Get-RoutingGroupConnector

List the Routing Group Connectors with their costs:

Get-RoutingGroupConnector | ft Name,Cost

For me the issue was that the Routing Group Connector set up for Exchange 2010 in Site 1 had a cost set to 100, whereas all other RGCs had a cost set to 1. This meant Site1 –> Site4 –> Site1 had a cost of 2. Site1 (2010) –> Site1 (2003) had a cost of 100. It’s easy to see why it won.

Using the following command I was able to set the costs to the same:

Set-RoutingGroupConnector -Identity "<Administrative Group>\<Routing Group>\<Connector>" -Cost 1

It’s quite difficult to get a grip on why the RGC cost takes precedence over the AD site cost, or the Exchange specific site cost – effectively you have 3 different settings to govern the mail routing. From my problem-solving steps, it would seem that The RGC takes precedence, followed by the Exchange specific cost, followed by the AD site link cost.

Any Exchange Guru out there care to correct or confirm?

Sam

Symptoms were:

• iPhone fails to sync, generic timeout error (or is very slow)
• https://www.testexchangeconnectivity.com/ successfully tests the mailbox access

The server was configured as per http://support.microsoft.com/kb/817379/en-us to allow OWA/ActiveSync with SSL on OWA.

The iPhone was configured to accept the SSL certificate on the Exchange Server.

My brother Tom sent me this Apple KB (http://support.apple.com/kb/TS3398) which he’d found from the other side – Exchange servers he was managing were under very heavy load, which is another symptom of this issue.

I installed the new configuration as per the article, restarted the phone and the issue was fixed!

• If

• you are using Microsoft System Center Operations Manager 2007

• you have a Microsoft Certificate Services 2003 Certificate Authority on your domain

• you have non-domain Windows Server 2008 servers you wish to monitor or set up as a gateway server.

Getting a certificate for either a Gateway Server or remotely monitored Server can be a touch vexing. If you’re installing on the same domain as the SCOM management server the security settings take care of themselves, not so for non-domain servers, which require mutual certificate authentication. The Gateway must trust the Domain CA and identify itself as trusted to the Management Server. I have bashed my head against this several times now, so I thought I’d make a precise blog post to cover the steps required!

In this scenario, we will have 2 servers CA01, the Windows 2003 Certificate Authority, and Gateway01, the SCOM 2007 gateway. The certificate template for Operations Manager has been created on CA01 as per the documentation and is called “OperationsManagerCert”. On Gateway01 I have copied the Gateway installer to c:\SCOM\Gateway and the SCOM Tools to c:\SCOM\Tools. SCOM01 is our SCOM collection server.

CA01: Navigate to https://ca01/certsrv and download the CA Certificate.

Gateway01: Copy the CA Certificate to the c:\SCOM folder by whatever means you have. Open mmc.exe and add the Certificates Snap-in for the local computer account. Right click the Trusted Root Certification Authorities store and Import the CA01 CA certificate.

Gateway01: Open notepad and create a new certificate request file with the contents below. Name the file Gateway01.inf and save in c:\SCOM

[NewRequest]

Subject="CN=<FQDN of Gateway01>"

Exportable=TRUE

KeyLength=2048

KeySpec=1

KeyUsage=0xf0

MachineKeySet=TRUE

[EnhancedKeyUsageExtension]

OID=1.3.6.1.5.5.7.3.1

OID=1.3.6.1.5.5.7.3.2

Gateway01: Open a command prompt as administrator and navigate to c:\SCOM, use certreq.exe to generate a certificate request:

certreq –new –f Gateway01.inf Gateway01.req

Gateway01: Open Gateway01.req in notepad and copy the contents to clipboard.

CA01: Open https://ca01/certsrv and start a new advanced certificate request, create the certificate request using a base64 encoded CMC. Paste the data from Gateway01.req into the “Saved Request” box. Select your SCOM certificate template and click next. Save the response as a Base 64 encoded certificate.

Gateway01: Copy the certificate file over to c:\SCOM on Gateway01 by whatever method you have available. Open a command prompt with admin rights and approve the new certificate with certutil.

certreq –accept Gateway01.cer

Check that the certificate has been imported into the Computer/Personal store using mmc.exe.

SCOM01: At this point you can either install your SCOM agent, or Gateway Server on Gateway01 – if you are installing the Gateway Server like me, you need to first approve the Gateway using the Gateway Approval Tool. Open a command prompt as administrator and navigate to “c:\Program Files\System Center Operations Manager 2007” or wherever your SCOM install is. Copy the Microsoft.EnterpriseManagement.GatewayApproval.Tool.exe from Support Tools into the parent folder (it requires .dlls in that folder).

Microsoft.EnterpriseManagement.GatewayApproval.Tool.exe ¬

/ManagementServerName=SCOM01 /GatewayName=Gateway01

Gateway01: Run the Gateway Server installer and enter the details of the Management Server and Management Group name. When that’s finished, you need to tell SCOM which certificate to use with the MOMCertImport.exe tool located in c:\SCOM\Tools

MOMCertImport /SubjectName Gateway01.Domain.Lcl

Give it a few minutes and you should be able to see the new gateway under Management Servers in the Administration console for SCOM. Remember to right-click, properties, security and allow the server to act as a proxy if it’s reporting for other servers.

I use the same procedure to install Agents in my DMZ that don’t have access to the certificate services – likewise our production web servers isolated in their hosting environment.

I hope this helps you, I know this is an article that I will be referring back to time and time again!

Fortunately this is very simple to counter – simply clear the old “Move Request” and the options will be back in the Mailbox options:

At some point I'll update to 1.6.1, but for now, I'm glad it's working again!

Similarly, when I tried to access the c$, ADMIN$ shares on the server, it would deny me access, and would lock out my admin account when I tried. Creating a separate share would allow me access, but that’s no good for PSExec. To further confuse things, when I accessed the \\server\c$ share from the server, it worked.

Checking the share properties using “net share c$” shows that the settings are all correct, Everyone has FULL access (this is default, it uses NTFS permissions to restrict access):

This issue does not affect domain member servers, I was able to browse to the c$ shares of several Windows Server 2008 servers on the domain.

The problem is caused by UAC and the elevated privileges required to access the administrative shares. This Microsoft KB article (951016) describes the issue in Windows Vista

To better protect those users who are members of the local Administrators group, we implement UAC restrictions on the network. This mechanism helps prevent against "loopback" attacks. This mechanism also helps prevent local malicious software from running remotely with administrative rights.

and the steps to resolve it, open a new PowerShell window as administrator:

New-ItemProperty HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\System -name "LocalAccountTokenFilterPolicy" -value "1" -propertyType dword

A word of caution: this is opening up a security hole and it should only be done with careful consideration of the risks. The need to use PSExec to remotely run a process was an important part of the deployment, however the same result could be achieved using PowerShell remoting. Until it’s tested and we’re ready to deploy that, I’ll be using this method.

I couldn’t manually migrate any VMs either – it was odd, because both hosts were added into the cluster, and could ping and vmkping each other from the console.

I also received email alerts -

[VMware vCenter - Alarm Host error] Error detected on [HOST] in [Data Center]: Agent can't send heartbeats.msg size: 1266, sendto() returned: Operation not permitted

It turns out that there were slight naming differences between the default VMKernels on each host, which stops communication. Since one VMKernel was named “VMKernel” and the other “VMKernel 2” it stops the migrations, and hence DRS. The hosts would add into the cluster OK, DRS actually showed as “imbalanced” on the Cluster summary screen - it was just DRS and vMotion which wouldn’t work.

With the VMKernels renamed to exactly the same thing, DRS kicked off no problem, as did a manual migration.

So the moral of the story is this; name ALL networks in the same cluster identically. It makes sense when you think that the VM needs to see it’s Virtual Machine Network on each host – why should the Service Console and VMKernel be any different?

The certificate specified in the registry at HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft Operations Manager\3.0\Machine Settings cannot be used for authentication.  The error is The credentials supplied to the package were not recognized(0x8009030D).

I followed the Microsoft install and setup guides exactly, and it’s not my first time either – but I’d never seen that one before.

It turns out that it’s a quirk with Certificate Services and how you request your certificate. I used the Certificate Services website on my Server 2003 Enterprise Root Certificate Authority to request the correct certificate, based on the OperationsManager template I created. Crucially, there wasn’t the option to import the certificate to the Machine/Personal certificate store – it went into the User/Personal. This meant that when it came to exporting and then re-importing the certificate, the private key was not correct.

Requesting the certificate through the MMC Certificates Snap-in and restarting the Health Service resolves the issue.

I’ve installed a new SQL Server 2008 server “SQL1” on Windows Server 2008 to take some of the load, and take advantage of the 64-bit OS and SQL installation.

Both servers are part of the domain “DOMAIN”

The process goes something like this:

1. Add the user that SIM runs as to the SQL server logins. For me that’s “DOMAIN\Insight.Manager”
2. Create a new database on SQL1 with exactly the same name as the MONITOR1 database for SIM. Since my 6.x install is an upgraded 5.x install, the database is called “Insight_v50_0_16732390”.
3. Add the SIM user account to the new database with DBO permissions.
4. Stop the HP SIM service on MONITOR1
5. Right click “Insight_v50_0_16732390” on MONITOR1 and Export. Export all the tables to SQL1…and wait a long time for the data to transfer.
6. While you’re waiting, you can edit the following files (c:\Program Files\HP\Systems Insight Manager\Config\) - database.props and database.admin. Change any references for MONITOR1 to SQL1.
7. Once it’s completed, stop the SQL server on MONITOR1 and start the HP SIM services again - fire up the SIM homepage to check everything is running OK.
8. If it all checks out, remove the old database and if it’s no longer needed, uninstall the SQL server too.

Reporting Services Error 
--------------------------------------------------------------------------------

The permissions granted to user 'MCGEOWN\Sam.McGeown' are insufficient for
performing this operation. (rsAccessDenied) Get Online Help

--------------------------------------------------------------------------------
SQL Server Reporting Services

Additionally, you may be able to access the http://SERVER/Reports site, but will have no permissions:

You may also spend a good while checking DB permissions, IIS configurations, file permissions and so on. And after all that, you may just stumble upon the fact that if you run Internet Explorer as Administrator, it will work as expected.

Event ID 1030 and 1058 every 5 minutes, looking into the detail for these events I can see its a replication issue for one of the GPOs.

Event Type:	Error
Event Source:	Userenv
Event Category:	None
Event ID:	1030
Date:		29/03/2010
Time:		04:01:29
User:		NT AUTHORITY\SYSTEM
Computer:	DC01
Description:
Windows cannot query for the list of Group Policy objects.
Check the event log for possible messages previously logged by the
policy engine that describes the reason for this.


For more information, see Help and Support Center at 

http://go.microsoft.com/fwlink/events.asp.

The slightly more informative 1058 showed

Event Type:	Error
Event Source:	Userenv
Event Category:	None
Event ID:	1058
Date:		29/03/2010
Time:		04:06:30
User:		NT AUTHORITY\SYSTEM
Computer:	DC01
Description:
Windows cannot access the file gpt.ini for GPO CN={3A7AC061-A26C-4154
-8CF5-01D5754E5C2C},CN=Policies,CN=System,DC=DOMAIN,DC=LCL.
The file must be present at the location <\\DOMAIN.LCL\SysVol\DOMAIN.LCL
\Policies\{3A7AC061-A26C-4154-8CF5-01D5754E5C2C}\gpt.ini>. (Access is denied. ).
Group Policy processing aborted. 

For more information, see Help and Support Center at 

http://go.microsoft.com/fwlink/events.asp.

There was no visible cause for the errors, DFS had just got it’s knickers in a twist somehow and was not resolving the DFS share for the domain correctly.

The resolution was fairly simple, running the “dfsutil /purgeMUPCache” command seems to have resolved it for now. The /PurgeMUPCache command clears the MUP Cache (duh!) which holds info about DFS and other shares on the client system.

Technet says: “Clears the client MUP cache, preventing confusion about the current provider when such names conflict. Except for a temporary performance hit, this command has no other adverse effects. This command does not affect any DFS metadata. If this command is not run, and the namespace is not accessed, the obsolete cache entry eventually expires.”

There are plenty of other causes for these errors, if your server is multi-homed (multiple NICs) then check that your “public” NIC is at the top of the adaptor bindings.

My DC is now running happily, no 1030 or 1058s.

In this post, I’m covering Installing the IIS web server (and a few useful bits) and managing it from the IIS Management Snap-in.

Installing the basic IIS installation

Installing optional components in Server 2008 R2 Core is handled by two commands, OCList and OCSetup. OCList, as the name suggests, lists the optional components and their status, installed or not installed. It’s a long list, so I recommend issuing the command with the “|more” pipe:

oclist | more

The output looks something like this:

OCSetup will accept any one, or multiple, of the roles listed in OCList as an argument to install. It’s recommended you use the command with “start /w” preceding so that the command prompt will wait for the installation to finish before continuing.

To install the basic IIS web server install, use

start /w ocsetup IIS-WebServerRole

As far as I can see, this installs the roles:

Installed:IIS-WebServerRole

Installed:IIS-WebServer

Installed:IIS-ApplicationDevelopment

Installed:IIS-CommonHttpFeatures

Installed:IIS-DefaultDocument

Installed:IIS-DirectoryBrowsing

Installed:IIS-HttpErrors

Installed:IIS-StaticContent

Installed:IIS-HealthAndDiagnostics

Installed:IIS-HttpLogging

Installed:IIS-Performance

Installed:IIS-HttpCompressionStatic

Installed:IIS-Security

Installed:IIS-RequestFiltering

Installed:IIS-WebServerManagementTools

In order to get .Net functioning and allow remote management, you’ll also need the following components installed, a registry key added and the Web Management Service Started (in order):

start /w ocsetup WAS-NetFxEnvironment
start /w ocsetup IIS-ISAPIExtensions
start /w ocsetup IIS-ISAPIFilter
start /w ocsetup IIS-NetFxExtensibility
start /w ocsetup IIS-ASPNET
start /w ocsetup IIS-ManagementService

reg add HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\WebManagement\Server /v EnableRemoteManagement /t REG_DWORD /d 1

net start wmsvc

You should now be able to manage your IIS server via the IIS Management Console on a Windows Server 2008 or Windows 7 PC with Remote Server Administration Tools installed.

You can also manage IIS through a PowerShell addin, if you run powershell.exe on your Server Core installation, then import the WebAdministration Module:

C:\Users\Administrator>powershell
Windows PowerShell
Copyright (C) 2009 Microsoft Corporation. All rights reserved.

PS C:\Users\Administrator> Import-Module WebAdministration
PS C:\Users\Administrator> Get-Command -PsSnapin WebAdministration

CommandType     Name                            Definition
-----------     ----                            ----------
Cmdlet          Add-WebConfiguration            Add-WebConfiguration [-Filte...
Cmdlet          Add-WebConfigurationLock        Add-WebConfigurationLock [-F...
Cmdlet          Add-WebConfigurationProperty    Add-WebConfigurationProperty...
Cmdlet          Backup-WebConfiguration         Backup-WebConfiguration [-Na...
Alias           Begin-WebCommitDelay            Start-WebCommitDelay
Cmdlet          Clear-WebConfiguration          Clear-WebConfiguration [-Fil...
Cmdlet          Clear-WebRequestTracingSettings Clear-WebRequestTracingSetti...
Cmdlet          ConvertTo-WebApplication        ConvertTo-WebApplication [[-...
Cmdlet          Disable-WebGlobalModule         Disable-WebGlobalModule [-Na...
Cmdlet          Disable-WebRequestTracing       Disable-WebRequestTracing [[...
Cmdlet          Enable-WebGlobalModule          Enable-WebGlobalModule [-Nam...
Cmdlet          Enable-WebRequestTracing        Enable-WebRequestTracing [[-...
Alias           End-WebCommitDelay              Stop-WebCommitDelay
Cmdlet          Get-WebAppDomain                Get-WebAppDomain [-InputObje...
Cmdlet          Get-WebApplication              Get-WebApplication [[-Name] ...
Cmdlet          Get-WebAppPoolState             Get-WebAppPoolState [[-Name]...
Cmdlet          Get-WebBinding                  Get-WebBinding [[-Name] <Str...
Cmdlet          Get-WebConfigFile               Get-WebConfigFile [[-PSPath]...
Cmdlet          Get-WebConfiguration            Get-WebConfiguration [-Filte...
Cmdlet          Get-WebConfigurationBackup      Get-WebConfigurationBackup [...
Cmdlet          Get-WebConfigurationLocation    Get-WebConfigurationLocation...
Cmdlet          Get-WebConfigurationLock        Get-WebConfigurationLock [-F...
Cmdlet          Get-WebConfigurationProperty    Get-WebConfigurationProperty...
Cmdlet          Get-WebFilePath                 Get-WebFilePath [[-PSPath] <...
Cmdlet          Get-WebGlobalModule             Get-WebGlobalModule [[-Name]...
Cmdlet          Get-WebHandler                  Get-WebHandler [[-Name] <Str...
Cmdlet          Get-WebItemState                Get-WebItemState [[-PSPath] ...
Cmdlet          Get-WebManagedModule            Get-WebManagedModule [[-Name...
Cmdlet          Get-WebRequest                  Get-WebRequest [-InputObject...
Cmdlet          Get-Website                     Get-Website [[-Name] <String...
Cmdlet          Get-WebsiteState                Get-WebsiteState [[-Name] <S...
Cmdlet          Get-WebURL                      Get-WebURL [[-PSPath] <Strin...
Cmdlet          Get-WebVirtualDirectory         Get-WebVirtualDirectory [[-N...
Function        IIS:                            set-location IIS:
Cmdlet          New-WebApplication              New-WebApplication [-Name] <...
Cmdlet          New-WebAppPool                  New-WebAppPool [-Name] <Stri...
Cmdlet          New-WebBinding                  New-WebBinding [[-Name] <Str...
Cmdlet          New-WebFtpSite                  New-WebFtpSite [-Name] <Stri...
Cmdlet          New-WebGlobalModule             New-WebGlobalModule [-Name] ...
Cmdlet          New-WebHandler                  New-WebHandler [-Name] <Stri...
Cmdlet          New-WebManagedModule            New-WebManagedModule [-Name]...
Cmdlet          New-Website                     New-Website [-Name] <String>...
Cmdlet          New-WebVirtualDirectory         New-WebVirtualDirectory [-Na...
Cmdlet          Remove-WebApplication           Remove-WebApplication [-Name...
Cmdlet          Remove-WebAppPool               Remove-WebAppPool [-Name] <S...
Cmdlet          Remove-WebBinding               Remove-WebBinding [-Protocol...
Cmdlet          Remove-WebConfigurationBackup   Remove-WebConfigurationBacku...
Cmdlet          Remove-WebConfigurationLocation Remove-WebConfigurationLocat...
Cmdlet          Remove-WebConfigurationLock     Remove-WebConfigurationLock ...
Cmdlet          Remove-WebConfigurationProperty Remove-WebConfigurationPrope...
Cmdlet          Remove-WebGlobalModule          Remove-WebGlobalModule [-Nam...
Cmdlet          Remove-WebHandler               Remove-WebHandler [-Name] <S...
Cmdlet          Remove-WebManagedModule         Remove-WebManagedModule [-Na...
Cmdlet          Remove-Website                  Remove-Website [-Name] <Stri...
Cmdlet          Remove-WebVirtualDirectory      Remove-WebVirtualDirectory [...
Cmdlet          Rename-WebConfigurationLocation Rename-WebConfigurationLocat...
Cmdlet          Restart-WebAppPool              Restart-WebAppPool [[-Name] ...
Cmdlet          Restart-WebItem                 Restart-WebItem [[-PSPath] <...
Cmdlet          Restore-WebConfiguration        Restore-WebConfiguration [-N...
Cmdlet          Select-WebConfiguration         Select-WebConfiguration [-Fi...
Cmdlet          Set-WebBinding                  Set-WebBinding [[-Name] <Str...
Cmdlet          Set-WebConfiguration            Set-WebConfiguration [-Filte...
Cmdlet          Set-WebConfigurationProperty    Set-WebConfigurationProperty...
Cmdlet          Set-WebGlobalModule             Set-WebGlobalModule [-Name] ...
Cmdlet          Set-WebHandler                  Set-WebHandler [-Name] <Stri...
Cmdlet          Set-WebManagedModule            Set-WebManagedModule [-Name]...
Cmdlet          Start-WebAppPool                Start-WebAppPool [[-Name] <S...
Cmdlet          Start-WebCommitDelay            Start-WebCommitDelay [-Verbo...
Cmdlet          Start-WebItem                   Start-WebItem [[-PSPath] <St...
Cmdlet          Start-Website                   Start-Website [[-Name] <Stri...
Cmdlet          Stop-WebAppPool                 Stop-WebAppPool [[-Name] <St...
Cmdlet          Stop-WebCommitDelay             Stop-WebCommitDelay [[-PSPat...
Cmdlet          Stop-WebItem                    Stop-WebItem [[-PSPath] <Str...
Cmdlet          Stop-Website                    Stop-Website [[-Name] <Strin...

Serve up a .NET page, to taste

Not that you’d doubt me (!) but there’s one last thing to do - prove it worked. Fortunately, there’s an easy way to do that. I borrowed the code from www.codefixer.com to create a little “hello world” page. Since the default website is c:\inetpub\wwwroot\ I saved the page there as default.aspx and fired up my browser – et voila!

I’m going to look at some management tasks – the bread and butter of being a Windows admin.

Activating Server 2008 Core

Activating Server 2008 Core is done via a pre-packaged script called slmgr.vbs -  “Windows Software Licensing Management Tool”

Firstly, you have to install a Product Key (unless it was done during your install)

cscript C:\windows\system32\slmgr.vbs /ipk <Product Key>

After that, it’s just a case of automatic activation, assuming you have internet access

cscript C:\windows\system32\slmgr.vbs /ato

If you’ve not got internet access for the server you can use the /dti option to get the Activation ID, call the Microsoft Licensing and Activation line and tap it in. Then use the /atp option to enter the response and activate.

Windows Updates

If you read the last post in this series, Configuring Server 2008 R2 Core Series: Network Settings, you may have seen the option in sconfig.cmd to set Windows Update settings. That’s the first, interactive, way to configure Windows Updates. It’s worth noting that the easiest way to do this is via your Group Policies, if you’re on a domain.

===============================================================================
                         Server Configuration
===============================================================================

1) Domain/Workgroup:                    Domain:  MCGEOWN.LOCAL
2) Computer Name:                       ServerCore2008
3) Add Local Administrator
4) Configure Remote Management

5) Windows Update Settings:             Manual
6) Download and Install Updates
7) Remote Desktop:                      Disabled

8) Network Settings
9) Date and Time

10) Log Off User
11) Restart Server
12) Shut Down Server
13) Exit to Command Line

Enter number to select an option: 5

Windows Update currently set to: Manual
Select (A)utomatic or (M)anual updates: A

Enabling Automatic updates...

The second method is the more command-line, scripting method. This sets it to download automatically and install at 3am every day (“/au 1” disables, “/au /v” shows current value):

Cscript c:\windows\system32\scregedit.wsf /au 4

Enabling Remote Management

Similarly to Windows Updates, remote management can be configured via sconfig.cmd or command line. Here’s how:

Enter number to select an option: 4
--------------------------------
  Configure Remote Management
--------------------------------

1) Allow MMC Remote Management
2) Enable Windows PowerShell
3) Allow Server Manager Remote Management
4) Show Windows Firewall settings

5) Return to main menu

Enter selection: 1

Enabling MMC firewall exceptions and Virtual Disk Service...

Enter selection: 2

Enabling Windows PowerShell...
Setting Windows PowerShell execution policy to remotesigned...

[Server requests a reboot here - you can't enable Server Manager until it's done]

Enter selection: 3

Setting Windows PowerShell execution policy to remotesigned...
Enabling Server Manager cmdlets...

Configuring Remote Server Manager settings...

If you need to do this via the command line, it happens like this…

Enable WinRM:

C:\Users\Administrator> winrm quickconfig
WinRM is not set up to allow remote access to this machine for management.
The following changes must be made:
Create a WinRM listener on HTTP://* to accept WS-Man requests to any IP on this machine.
Make these changes [y/n]? y
WinRM has been updated for remote management.
Created a WinRM listener on HTTP://* to accept WS-Man requests to any IP on this machine.

Fire up PowerShell (powershell.exe) and set the execution policy to RemoteSigned

Set-ExecutionPolicy RemoteSigned

Then enable the Remote Administration rules on the firewall:

netsh advfirewall firewall set rule group="Remote Administration" new enable=yes

To configure management by Remote Desktop, you can run the now-familiar sconfig.cmd and select option 7, or you can issue the following commands:

cscript c:\windows\system32\scregedit.wsf /ar 0

netsh advfirewall firewall set rule group="Remote Desktop" new enable=yes

Bear in mind that your RDP session will need TLS authentication and will not give you a desktop or GUI, just the command line interface. If you need to disable TLS for older clients (e.g. XP) you have to disable it with the following command:

cscript C:\Windows\System32\Scregedit.wsf /cs 0

Can you manage?

With all those steps completed, you should be able to connect to your server with Remote Server Administration Tools on any Server 2008 or Windows 7 computer.

This is my Windows 7 PC connected via “Server Manager”

To connect via WinRS (Windows Remote Shell) and execute remote commands, use:

winrs -r:<server name> <command>

e.g:

winrs –r:<Server Name> cmd

Allows me access to the command shell on that server.

Finally, this is what RDP to the same server looks like:

Hopefully that gives you a few options for managing your Windows Server 2008 Core machine!

Using the Server Configuration Tool

The server configuration tool (sconfig.cmd) is provided in R2 for some of the basic setup tasks, so you can run that by issuing the “sconfig” command. Out of the box, it looks something like this:

As you can see, this interactive tool will step you through configuring the network settings (Option 8), Computer Name (Option 2) or Domain/Workgroup (Option 1).

Enter number to select an option: 8

--------------------------------
    Network settings
--------------------------------

Available Network Adapters

Index#  IP address      Description

  0     192.168.8.117   Intel(R) PRO/1000 MT Network Connection

Select Network Adapter Index# (Blank=Cancel):  0

--------------------------------
    Network Adapter Settings
--------------------------------

NIC Index               0
Description             Intel(R) PRO/1000 MT Network Connection
IP Address              192.168.8.117
Subnet Mask             255.255.255.0
DHCP enabled            True
Default Gateway         192.168.8.1
Preferred DNS Server    192.168.8.5
Alternate DNS Server    192.168.8.22

1) Set Network Adapter IP Address
2) Set DNS Servers
3) Clear DNS Server Settings
4) Return to Main Menu

Select option:  1

Select (D)HCP, (S)tatic IP (Blank=Cancel): S
Set Static IP
Enter static IP address: 192.168.8.220
Enter subnet mask (Blank = Default 255.255.255.0):
Enter default gateway: 192.168.8.1
Setting NIC to static IP...

--------------------------------
    Network Adapter Settings
--------------------------------

NIC Index               0
Description             Intel(R) PRO/1000 MT Network Connection
IP Address              192.168.8.220
Subnet Mask             255.255.255.0
DHCP enabled            False
Default Gateway         192.168.8.1
Preferred DNS Server
Alternate DNS Server

1) Set Network Adapter IP Address
2) Set DNS Servers
3) Clear DNS Server Settings
4) Return to Main Menu

Select option:  2
DNS Servers

Enter new preferred DNS server (Blank=Cancel): 192.168.8.22
Enter alternate DNS server (Blank = none): 192.168.8.5
Alternate DNS server set.

--------------------------------
    Network Adapter Settings
--------------------------------

NIC Index               0
Description             Intel(R) PRO/1000 MT Network Connection
IP Address              192.168.8.220
Subnet Mask             255.255.255.0
DHCP enabled            False
Default Gateway         192.168.8.1
Preferred DNS Server    192.168.8.22
Alternate DNS Server    192.168.8.5

1) Set Network Adapter IP Address
2) Set DNS Servers
3) Clear DNS Server Settings
4) Return to Main Menu


Select option:  4

Enter number to select an option: 2

Computer Name

Enter new computer name (Blank=Cancel): SERVERCORE2008
Changing Computer name...

Enter number to select an option: 1

Change Domain/Workgroup Membership

Join (D)omain or (W)orkgroup? (Blank=Cancel) D

Join Domain
Name of domain to join:  MCGEOWN.LOCAL
Specify an authorized domain\user:  MCGEOWN\sam.mcgeown

Joining MCGEOWN.LOCAL...

Enter the password of the authorized user:

Command Line Configuration with Netsh/Netdom

There’s also a manual method (e.g. for a scripted installation and config) using Netsh and Netdom commands that most Windows admins will be familiar with.

List the interfaces (network adaptors):

netsh interface ipv4 show interfaces

Identify the name of the interface you want to assign an IP for and configure:

netsh interface ipv4 set address name="<Interface Name>" source=static address=<IP Address> mask=<Subnet Mask> gateway=<Gateway>

Configure DNS servers:

netsh interface ipv4 add dnsservers "<Interface Name>" <DNS Server IP> index=<number>

If you want to add more than one IP address for your server, try:

netsh interface ipv4 add address name="<Interface Name>" address=<Additional IP> mask=<Subnet Mask>

To change your computer’s name, you can use (leave off the /reboot if you don’t want to yet):

netdom renamecomputer /newname:<New Name> /reboot

To join your computer to a domain, you can use (leave off the /reboot if you don’t want to yet):

netdom /join /domain:<domain> /UserO:<domain\user to join with> /PasswordO:<Password> /reboot

Not so different after all?

At the end of all that, you can see that configuring basic network settings with sconfig.cmd is pretty straight forward, and configuring basic network settings for scripts, or a more command-line based admin, is also quite do-able.

C:\Users\Administrator>ipconfig /all

Windows IP Configuration

   Host Name . . . . . . . . . . . . : ServerCore2008
   Primary Dns Suffix  . . . . . . . : MCGEOWN.LOCAL
   Node Type . . . . . . . . . . . . : Hybrid
   IP Routing Enabled. . . . . . . . : No
   WINS Proxy Enabled. . . . . . . . : No
   DNS Suffix Search List. . . . . . : MCGEOWN.LOCAL

Ethernet adapter Local Area Connection:

   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : Intel(R) PRO/1000 MT Network Connection
   Physical Address. . . . . . . . . : 00-50-56-AB-28-8B
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes
   Link-local IPv6 Address . . . . . : fe80::3c4d:cdd1:5a4a:fbff%3(Preferred)
   IPv4 Address. . . . . . . . . . . : 192.168.8.220(Preferred)
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   IPv4 Address. . . . . . . . . . . : 192.168.8.221(Preferred)
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   IPv4 Address. . . . . . . . . . . : 192.168.8.222(Preferred)
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   IPv4 Address. . . . . . . . . . . : 192.168.8.223(Preferred)
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   Default Gateway . . . . . . . . . : 192.168.8.1
   DHCPv6 IAID . . . . . . . . . . . : 50352214
   DHCPv6 Client DUID. . . . . . . . : 00-01-00-01-13-3B-B4-6C-00-50-56-AB-28-8B

   DNS Servers . . . . . . . . . . . : 192.168.8.5
                                       192.168.8.22
   NetBIOS over Tcpip. . . . . . . . : Enabled

C:\Users\Administrator>

Netsh command reference | Netdom command reference

Windows Server 2008 Active Directory, Configuration

Windows Server 2008 Applications Infrastructure, Configuration

Windows Server 2008 Network Infrastructure, Configuration

Next I’ll be looking at the 70-647 to get the full MCITP: Enterprise Administrator (I already took the 70-620 exam for my MCSE).

Here’s how to use TCPDUMP by Micro Olap to extend that functionality to your Windows boxes.

Firstly you need to find the interface number of the network adaptor you are trying to find CDP data for.  Use this command:

tcpdump -D

Which gives you a list of the interfaces on the computer:

My actual NIC is the third one in the list, so I can run the command:

tcpdump -i 3 -nn -v -s 1500 -c 1 ether[20:2] == 0x2000

-i n [interface and the number in the list, for me 3]

-nn [don’t resolve DNS, speeds things up]

-v [verbose mode, otherwise we won’t see all the packet details]

-s 1500 [set the maximum packet size to capture, the MTU is 1500 by default so it will capture the entire packet]

-c 1  [Capture one packet only, since we only want the CDP packet and filter using the header]

ether[20:2] == 0x2000 [Check the Ethernet header packet ID for the hex value 0x2000 – CDP protocol]

Some output is omitted, but you can see that the name of the switch and the port are both in there.

Easier than tracing a cable!

Here’s how to enable it:

Upgrade the VM’s tools, if you haven’t already. This requires a re-boot, but don’t reboot, shut the server down down, otherwise you’ll require another reboot to add the feature. The tools can be upgraded from the system tray icon.

Once the VM is shut down, upgrade the Virtual Hardware (again, assuming you haven’t already!)

Once your virtual machine is fully up to date with VMTools and Hardware, you can edit the Advanced options (again, while the machine is shut down) and enable the feature.

Now with the server booted, you can add Memory and CPU without the need for a reboot. In my tests changing the Memory from 1GB to 2GB caused a single dropped ping, and changing from 1 vCPU to 2 vCPUs caused a ping to drop.

I’m going to be upgrading a Server 2008 R2 x64 SP2 Standard Edition virtual server to R2. To see what editions can and can’t be upgraded, check out this Technet Article, but it’s safe to say that you can’t upgrade across architectures (32-bit to 64-bit) and you can’t downgrade SKUs (Enterprise to Standard).

The first step, as ever, is always to back up your server, if the upgrade goes wrong, you can always restore and try again. You have been warned!

So, without further ado, slip in your R2 DVD and begin…

Install

Update

Select your target SKU

Select “upgrade” (obviously

Check your upgrade report (which is saved as HTML on your desktop. The first time I ran this it said that I didn’t have enough free space – it required a whopping 15GB, which makes me think that this is no Server 2003 –> R2 upgrade, it’s the full blown OS install. Assuming everything checks out, go ahead.

Sit back and grab a cup of coffee. After a while, you’ll reboot

  and the upgrade begins in earnest. Once the process is completed, and another reboot has happened, you’ll be upgraded to R2. You’ll need to activate it with your R2 key.

Once you’re activated, update your server using Microsoft update or your patching method.

Et voila!

I do consider the Cisco qualifications as significantly more valuable than the others that I hold, simply because of the difficulty of the exams. I do find them “honest” in that they’re not trick questions, and you don’t need a technique to pass – just in depth knowledge.

Anyway, I think I’ll take few weeks before I look to my next study/exam.

After testing on my local IIS 7 and working perfectly, I uploaded the updates to my live blog and hit the “Migrate to IIS 7” button, which promises it will be completed in 24h. I received the “update your DNS” email, and duly updated my A records to the new server, and the transfer seems to be ok – aside from the fact that viewing any specific post causes an error – I’m guessing with the permissions of the App_Data folder. The catch being that I can’t access my IIS settings until GoDaddy have completed their 24h migration process.

It’s now been more than 72 hours since I kicked of the migration and still I cannot access and fix the IIS permissions issue which is dogging my blog. I’ve emailed twice and am still waiting for some resolution. Perhaps I won’t be renewing this year?

Note – this configuration will work for ESXi 4 upwards due to the server 2008 MSCS requirement for persistent SCSI-3 reservations.

The first step is to create a new vSwitch for the host-only cluster heartbeat network, don’t assign any network adaptors to the switch as it’s going to be local only.

Create a new virtual machine with a single hard disk. For the purposes of this test, I’ve assigned 2 vProcessors, 1GB RAM, 30GB drive for the OS, 1 vNIC in the default vSwitch0.

Add a second vNIC and assign it to the cluster network vSwitch created in step 1.

Install Windows Server 2008 R2 Enterprise and all the Windows Updates, for the example I’ve named it SQLCluster01.

Clone the server and rename the new one to SQLCluster02. In ESXi you can’t clone, so shut down the first server, copy the files to a new folder and right click the VMX file to add it to the inventory. When you boot it the first time VMware will ask if it’s been moved or copied – select copied.

Create a disk for use as the Quorom, this needs to be shared and since I’m using ESXi with local storage only it must be “eagerzeroedthick”. To do this I have to use the unsupported mode in ESXi (Alt+F1, type unsupported and then your root password) and use the vmkfstools command to create it (vmkfstools –c <size> –d eagerzeroedthick –a lsilogic /vmfs/volumes/<datastore>/<folder>/<disk>.vmdk)

Add the new disk to SQLCluster01 using a new SCSI virtual controller (different from the current controller, e.g. my first HD is on SCSI 0:1, the Quorum is on SCSI 1:0)

Check that the new SCSI controller is set to LSILogic (it is for Server 2008 by default) and set the SCSI Bus Sharing to Virtual.

Add the Quorum disk to the second virtual machine, using the same settings.

Edit the .vmx file for both servers, adding in the following lines (edit for your SCSI controller):

scsi1:0.mode = "independent-persistent"
scsi1:0.shared = "TRUE"

Create a disk for some shared storage for the cluster too, it will be needed for the DTC application as well as the SQL server – in a production environment you may want to separate logs and data, but for my test, I’m just adding another two 10GB disks. Use the same process as for creating the Quorum disk.

Configuring SQLCluster01 and SQLCluster02 for Microsoft Cluster Services

Once the two VMs are booted, we need to do a number of things:

Activate and format the Quorum disk. In SQLCluster01, open the disk management snap-in and activate the new Quorum disk, create a new partition and format it to NTFS. I mounted it at Q. In SQLCluster02, open the snap-in and activate the Quorum disk – it should pick up the partition and formatting automatically. I re-assigned the disk letter to Q.

Configure the Access Network. I’ve assigned a static IP within the servers range, 192.168.8.0/24 for both servers. They have full network connectivity.

Configure the Heartbeat Network. I’ve assigned the IPs 10.0.0.1 and 10.0.0.2 on a /29 subnet to the Cluster Heartbeat Network Adaptors, and renamed it to Cluster Heartbeat just to keep it tidy.

Join the domain. You can no longer use a workgroup for MSCS, so join the VMs to your domain.

Create DNS records for the Cluster, Virtual-SQL and MSDTC. An A record is required for the cluster itself, the virtual SQL instance and the Distributed Transaction Coordinator. I created vm-cluster, vm-virtualsql and vm-msdtc each with their own IPs.

Install Failover Clustering on both nodes. Launch the “Add Features” wizard and tick to install the Failover Clustering feature for each server (the server may require a restart).

Configure the Cluster. Open the Failover Cluster Management console from Administrative Tools and run the “Validate a Configuration” wizard. Add both servers into the wizard and run all the defaults.

Create the cluster. Run the “Create a Cluster…” wizard, select the servers as you did with the validation wizard, enter the cluster name and IP address that you created earlier (vm-cluster for me). Review the summary and create the cluster. The cluster service should automatically select the best disk for the quorum.

Installing the Distributed Transaction Coordinator on the MSCS

Open the Failover Cluster Management console and right click Services and Applications and select “Configure a service or application”. Select the DTC and click next.

Enter the DNS name you configured earlier for the DTC and the IP address you assigned and click next.

Select the shared storage you’ve assigned to the cluster, and click next.

Complete the wizard and then verify that the DTC is running by selecting it in the Services and Applications window – the Server Name, MSDTC and Drive should all be online.

Installing SQL Server 2008 on a Windows Server 2008 cluster

Assuming you updated the server earlier, you should already have the .Net Framework 3.5 SP1 which is a pre-requisite for SQL Server 2008. I’ll move on to the installation at this point, so open the installed and select a “New SQL Server failover cluster installation”.

Click through the wizard, enter key and license info and install the Setup Support Files. Check and resolve any issues in the setup support rules.

Select the features that you require

Configure the SQL Server Network Name as you created earlier in the DNS records. I’m installing the default instance of SQL.

Check the disk space requirements are OK and click next. Click next to create a new Cluster Resource Group for SQL.

Select the available cluster disk and click next.

Configure the network address you configured earlier for the SQL Cluster

In the Cluster Security Policy dialog box, accept the default value of Use service SIDs (Microsoft’s recommended option). Configure your service accounts and collation, DB Engine Authentication and Data Directories. Again, for production you wouldn’t use the same disk for Logs, Data, TempDB etc, but this is just a test setup.

After that, click through to your installation and complete.

Verify all the resources required for the cluster application are available in the Failover Cluster Management console.

Once that’s all set up, it’s time to install the second cluster node. Run the SQL 2008 installer and select the “Add node to a SQL Server failover cluster” option. As before, go through the setup support rules, check the Cluster Node Configuration, configure the Service Accounts, step through the verifications and install the node.

If it all completed successfully, you now have a working Windows Server 2008 Cluster with an active/passive SQL 2008 database cluster, running on top of a single ESXi 4 Server. I tested connecting with SQL Manager to vm-virtualsql which works as expected. I also tested failing over between hosts with the Cluster console, again it worked as expected.

Thanks for reading and I hope this helps!

I spend perhaps 8 hours a day at my work PC, maybe 2 hours a day on my home laptop and my iPhone is with me pretty much 24/7 – all of which are both data sources, and data endpoints. They all remind me to do things. To add a bit more complication to the mix, some things are personal, some things are work related.

So, to summarise, I want email, calendaring and to-do/tasks on my desktop, laptop and iPhone, and I want to be able to add/edit/delete for any of them.

[more]

Step 1 – Email

My personal email is downloaded by POP3 to Gmail from my ISP’s (GoDaddy) email server. I use Outlook 2007 on both my Laptop and Desktop to connect via IMAP to my Gmail account. Both use my ISP’s SMTP server to send email. I also configured Gmail to send via GoDaddy's SMTP server, this allows me to send from my personal address rather than my Gmail address. Email is accessible from my iPhone via the Exchange server protocol (Gmail Sync). Since all of these access email on the Gmail storage, when an email is deleted/moved/replied to on any platform, it stays up to date.

Step 2 – Calendar

Once again Google is the central repository for the data, using Google Calendar Sync to synchronise my calendar on both my Laptop and Desktop Outlook. On my Desktop, Google Calendar Sync updates the corporate Exchange account. Again the iPhone calendar syncs over the Gmail Sync/Exchange protocol to Google directly.

Step 3 – To-do/Tasks

This one is the most difficult and I’ve not yet resolved it fully. Google do have a Tasks app, but it doesn’t have a sync tool. My corporate Exchange server has tasks, but I have no way of syncing it with my Laptop. At the moment I am using the Exchange tasks which is obviously sync’d with my Desktop Outlook. I’m also using a free app called IMLite on the iPhone to access the tasks on the Exchange, but it’s read only.

It’s easier to view a diagram!

Other things to note

• All the connections are over SSL, so they’re secure – that’s really important because it’s personal information and you don’t need just anyone getting it!
• I chose Gmail over other online hosts because of the storage (over 7.4GB and growing), because it hosts my calendar and tasks, and is easier to set up to SEND email from my SMTP server.
• I know Gmail is ad supported – but if you access via IMAP/Exchange protocols, you’ll never see them.
• I’d like to be using Google tasks and sync them with my Outlook, but as yet I’ve not found a way to do this (c’mon Google, release the app!)
• My iPhone is sync’d to my Laptop via iTunes, but only for media and contacts.

Finally, I’m looking at my options for photo sync (or online storage) but it’s got to be high res, I’m also looking at document sync, but I’m pretty sure Google has that nailed too. I much prefer having it all under one roof.

Any comments, ideas, suggestions, drop a comment below!

I’m really pleased to say that neglecting this site paid off, or rather the study did – I passed with a score of 930! It was a LOT harder than I had expected, I thought I’d walk out after 20m! It does now mean that I am CCENT. I’ll be taking the ICND2 exam early in the new year which will move me up to CCNA.

Also in the exams category, I’m taking a beta exam “PRO: Design & Deploy Messaging Solutions with Microsoft Exchange Server 2010”. Another snappy title and another bundle of fun!

Sam

So to start, I'll outline the process:

• Download the vSphere vCenter 4 installer from VMware (~1.8GB).
• Download your updated licensing for vSphere.
• Back up your VirtualCenter server.
• Run the installation.

I'm not going to run through the download of the installer or licensing, if you're not sure how to do that, probably best not to do the rest.

Backing up VirtualCenter Server

My VirtualCenter server is installed on a Virtual Machine, so this makes things a lot simpler – I'll just take a snapshot to start. Being a belt-and-braces kind of situation (live HA cluster), I'm also going to do the database and configuration backup too.

Databases - I'm using SQL Server 2005 express which is supported for vSphere vCenter, so there will be no database upgrades, however the schema will be changed. First off, I've connected to SQL with SQL Management Studio and run a full backup. As I have VMware Update Manager installed too, I'm backing up that database as well.

Configuration file – Make a copy of your vpxd.cfg file, which is stored in the C:\Documents and Settings\All Users\Application Data\VMware\VMware VirtualCenter folder.

SSL Certificates – In the same folder as the vpxd.cfg file there's a folder called SSL, which you'll want to backup too.

If you're not using integrated authentication for the database access, you need to ensure you have the user name and password for the DB access.

Once all that's gathered together and safely backed up, you can move on to the installation.

Installing vSphere vCenter

Open services.msc and stop the VMware VirtualCenter Server service.

Insert your vCenter installation CD, the installer pops up:

Click vCenter Server.

Notice it's detected the earlier version of vCenter server and is going to upgrade.

  Enter license details.

Enter DB user details, or leave blank if you're using integrated authentication like me.

If you have any plug-ins installed (e.g. VirtualCenter Update Manager, or Converter) it will let you know that they need to be up to date too.

Select to upgrade the vCenter Server database, and tick that you've backed up the database and SSL folders.

  Select the account that you want to use to run the vCenter Server service.

Configure some ports, I've left them as defaults.

  Finally, install. It will run a DB upgrade, and various other uninstall/upgrades.

At this point I sat and waited…and waited…and waited. SQL server was chewing 70-80% processor, it was progressing, just slowly.

Eventually, it finished and the server settled down. I ran through the upgrade of Update Manager and Converter Enterprise, all click and go.

Stage 1 complete!

This can be resolved by following the steps in MS KB 896861

HTH,

Sam

The error occurs because the virtual disk cannot be moved without moving the source files, the .vmx, .vswap etc. Simply drag the entire VM, rather than the virtual disk to the new Datastore.

If you're trying to move a 2nd, 3rd or nth disk and you get this error, drag the entire VM as per above over to the new Datastore, once that's completed, go back in to SVMotion and drag the whole VM across again, only this time before you apply, drag the nth disk back to the new Datastore.

I ran into a slightly confusing problem today - our SQL servers are all created with 4 disks on 4 separate LUNs (System, Swap, SQL Data and SQL Logs). When viewing the server through Virtual Center I couldn't see all of the LUNs, just the System LUN. It's not a major problem as the VM can see the storage, but a little annoying when you have to remember what LUN the disks are on.

Slightly more distressing was the fact that the System-LUN was running out of space - fast. A LUN that should have had about 150GB free was running dangerously low. On investigation I found various snapshot files were being stored in with the System-LUN, which is where the VM's VMX, vswap etc are situated. These were the snapshot delta files of the additional disks, which were on other storage! This isn't first apparent at first as the disk snapshots have been named sequentially by ESX, so a VM with 4 disks on separate LUNs will in fact create 4 snapshot files on the SYSTEM-LUN named VM01-00001.vmdk, VM01-00002.vmdk, VM01-00003.vmdk and VM01-00004.vmdk. 00001 is for the System disk, 00002 is for the Swap disk etc etc. This means that the IO on that LUN has been multiplied, and the storage space is shrinking very rapidly.

A little more digging and it seems that this is by design - snapshots are not meant to be kept for very long, and I think VMware made a deliberate decision to make it difficult to do so. Any virtual disks created for a VM, lets call it VM01, were named VM01.vmdk. When additional virtual disks were created through vCenter on a different LUN, they were still named VM01.vmdk - there's no conflict because they're in different locations. However, when vCenter takes a snapshot it places them with the original disk, and because it's got the same name as the existing disk it starts to enumerate them.

This is bad for a number of reasons - most prominent of which is that if the snapshot file grows large, vCenter does not handle the commit well. In fact, neither does ESX, but I'll get to that. vCenter will time out on any operation that takes more than 15 minutes, so a commit of a 10GB snapshot will look for all intents and purposes in vCenter like it's failed. On top of that, the enumeration of snapshot delta files can cause confusion as to which disk it actualy belongs to, and if that happens, commiting

We all know snapshots are performance killers, but the functionality they provide is not insignificant, and as with most things a balance has to be struck between the functionality and the performance.

So the headlines

• VMs created with disks on multiple LUNs in vCenter use the SAME DISK NAME (eg; for VM01 the disks were created in /vmfs/volumes/SYSTEM-LUN/VM01.vmdk, /vmfs/volumes/SWAP-LUN/VM01.vmdk etc etc).
  • Mitigate this by creating disks using the vmkfstools and adding them to the VM or renaming the existing disks (see below).
• Snapshots cause ALL disk delta files onto the "system" LUN (i.e. where your VMX file is stored.) This is bad because a) it multiplies your I/O on that disk and b) you negate the benefits of storing on multiple LUNs.
  • Mitigate this by deleting your snapshots. There's no other way*, don't try manually moving them or you will have problems.
• Commiting large snapshots takes time - LOTS of time - and can have a big performance hit on your server.
  • Mitigate this by shutting down your VM first and commiting the disk using the vmware-cmd out of business hours. You can also merge the old disk and snapshots into a "new" disk, then shut down the VM and boot with the "new disk".
• vCenter has a hard coded 15m timeout.
  • If you are doing a operation that will take longer than that, do it via the console!

* when I say there's no other way, I mean, there's no other practical way. There are methods to move the snapshot files to another LUN but they bring some serious problems with them.

Create a vmdk (virtual disk) using vmkfstools

• Log in to your server console.
• Type su - (to log in as root, enter root password, note the "-" to load the root user environment variables)
• Navigate to the storage that you wish to use. E.g. cd /vmfs/volumes/System-LUN/
• Create a new folder for the virtual disk: mkdir VM01
• Navigate to the folder: cd VM01
• Create the disk: vmkfstools -c <size> <filename> -a <buslogic|lsilogic>
• For help just type vmkfstools

Rename vmdk files using vmkfstools

• Shut down the VM in vCenter.
• Edit the VM settings and remove the disk you wish to change. Do not delete the file!
• Log in to your server console.
• type su - (to log in as root - enter root password, note the "-" to load the root user environment variables)
• use the command: vmkfstools -E /vmfs/volumes/<LUN>/<Server Name>/<Disk Name>.vmdk /vmfs/volumes/<LUN>/<Server Name>/<New Disk Name>.vmdk

• Go back to the vCenter and re-add the disk, using the new name.

Commit your snapshots using vmware-cmd

• Log in to your server console.
• Type su - (to log in as root, enter root password, note the "-" to load the root user environment variables).
• Use the vmware-cmd -l command to list your VMs. Note the path to the VM you want to deal with.
• Remove all snapshots for a VM: vmware-cmd /path/to/vm/VM01.vmx removesnapshots

Configure ESX server's vSwitch

Configuring the vSwitch is actually pretty simple, but there are a couple of gotchas, so don't skip this bit! First thing to note is that if you are making changes to the vSwitch and the Service Console is on that vSwitch you can quite easily lock yourself out. Make sure you configure this correctly, first time! In this setup, I am adding all 4 NICs to vSwitch0, which will be the only vSwitch. I'll then use Port Groups to assign VLANs and Active/Passive configurations to the VMKernel/Service Console.

First things first then - assign the four NICs to the vSwitch. This is done in the Configuration Tab in VMware Infrastructure Client, then the Networking page. Edit the properties of your vSwitch, then select the Network Adaptor tab. Add all the NICs you wish to team in there (they may already be in there, depending on your setup). You should end up with something that looks like this (note that I've not assigned any VLAN yet):

Now you need to configure the NIC teaming, so edit the vSwitch Properties and under the Ports tab select the vSwitch. Click edit, and then go to the NIC teaming tab. Configure the teaming options like this:

That's the easy part over and done with! Time to move onto the Cisco!

Configuring the Cisco Core Switch

Firstly, we need to log on to the switch and enter enable mode; I'm going to assume you know how to do this - if not, you really shouldn't be attempting this setup!

Determine the switches trunk load balancing setup by using the command "show etherchannel load-balance". It should look something like this:

If the protocol is NOT src-dst-ip, then you won't be able to establish a trunk connection with the ESX server. If your protocol is not src-dst-ip, change it with the command "port-channel load-balance src-dst-ip". This now matches the "Route based on IP hash" setting you configured in ESX. Although ESX has a setting for MAC based hashing, as does the Cisco, I was unable to get it to work.

Moving on. You need to create a Port-Channel interface for the trunk (this is a virtual interface that binds the 4 GigabitEthernet interfaces together). As i've got other Port-channels in use for connections to other switches, I'm setting up port-channel 40. Move to config mode (conf t) and then enter the setup:

interface Port-channel40
 description VMTEST01 Aggregate
 switchport trunk encapsulation dot1q
 switchport trunk native vlan 8
 switchport mode trunk
 switchport nonegotiate
 spanning-tree portfast trunk
end

Description simply adds a description, "switchport trunk encapsulation dot1q" sets the encapsulation of the trunk to 802.1Q. "switchport trunk native vlan 8" means that any traffic without a VLAN tag will be automatically assigned to VLAN 8. "switchport mode trunk" obviously designates that we want a trunk, rather than access. "switchport nonegotiate" means that it will not attempt to negotiate the protocol, and be a static trunk, rather than LCAP or PGaP. "spanning-tree portfast trunk" causes a Layer 2 LAN interface configured as an access port to enter the forwarding state immediately, bypassing the listening and learning states (i.e. if the link goes down and then comes back up, it will do so quickly).

With the Port-channel configured, you now need to edit your GigabitEthernet ports and assign them to the Port-channel. For each port in the trunk, enter the following config (this example is port 8 on the master switch in my stack, hence 1/0/8):

interface GigabitEthernet1/0/8
 description VMTEST01 VMNIC1
 switchport trunk encapsulation dot1q
 switchport trunk native vlan 8
 switchport mode trunk
 switchport nonegotiate
 channel-group 40 mode on
 spanning-tree portfast trunk
end

The difference between that and the Port-channel setup? "channel-group 40 mode on" is simply assigning the port-channel in static mode.

Once all four NICs are assigned you might have to wait a few minutes for every layer of the connection to settle down before the trunk comes up. To check the status of the etherchannel you can use the command "show etherchannel 40 summary", replacing the 40 for whichever number you assigned to your port-channel.

I hope this helps navigate the minefield that I found to be setting up the NIC teaming!

The error looked something like this:

P:\>dcdiag /test:dns

Domain Controller Diagnosis

Performing initial setup:
   Done gathering initial info.

Doing initial required tests

   Testing server: SITE\DC01
      Starting test: Connectivity
         ......................... DC01 passed test Connectivity

Doing primary tests

   Testing server: SITE\DC01

DNS Tests are running and not hung. Please wait a few minutes...

   Running partition tests on : ForestDnsZones

   Running partition tests on : DomainDnsZones

   Running partition tests on : Schema

   Running partition tests on : Configuration

   Running partition tests on : DOMAIN

   Running enterprise tests on : DOMAIN.com
      Starting test: DNS
         Test results for domain controllers:

            DC: DC01.DOMAIN.COM
            Domain: DOMAIN.com

               TEST: Forwarders/Root hints (Forw)
                  Error: Root hints list has invalid root hint server: a.root-se
rvers.net. (198.41.0.4)
                  Error: Root hints list has invalid root hint server: b.root-se
rvers.net. (192.228.79.201)
                  Error: Root hints list has invalid root hint server: c.root-se
rvers.net. (192.33.4.12)
                  Error: Root hints list has invalid root hint server: d.root-se
rvers.net. (128.8.10.90)
                  Error: Root hints list has invalid root hint server: e.root-se
rvers.net. (192.203.230.10)
                  Error: Root hints list has invalid root hint server: f.root-se
rvers.net. (192.5.5.241)
                  Error: Root hints list has invalid root hint server: g.root-se
rvers.net. (192.112.36.4)
                  Error: Root hints list has invalid root hint server: h.root-se
rvers.net. (128.63.2.53)
                  Error: Root hints list has invalid root hint server: i.root-se
rvers.net. (192.36.148.17)
                  Error: Root hints list has invalid root hint server: j.root-se
rvers.net. (192.58.128.30)
                  Error: Root hints list has invalid root hint server: k.root-se
rvers.net. (193.0.14.129)

               TEST: Dynamic update (Dyn)
                  Warning: Dynamic update is enabled on the zone but not secure
DOMAIN.com.

         Summary of test results for DNS servers used by the above domain contro
llers:

            DNS server: 128.63.2.53 (h.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.12
7.in-addr.arpa. failed on the DNS server 128.63.2.53

            DNS server: 128.8.10.90 (d.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.12
7.in-addr.arpa. failed on the DNS server 128.8.10.90

            DNS server: 192.112.36.4 (g.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.12
7.in-addr.arpa. failed on the DNS server 192.112.36.4

            DNS server: 192.203.230.10 (e.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.12
7.in-addr.arpa. failed on the DNS server 192.203.230.10

            DNS server: 192.228.79.201 (b.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.12
7.in-addr.arpa. failed on the DNS server 192.228.79.201

            DNS server: 192.33.4.12 (c.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.12
7.in-addr.arpa. failed on the DNS server 192.33.4.12

            DNS server: 192.36.148.17 (i.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.12
7.in-addr.arpa. failed on the DNS server 192.36.148.17

            DNS server: 192.5.5.241 (f.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.12
7.in-addr.arpa. failed on the DNS server 192.5.5.241

            DNS server: 192.58.128.30 (j.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.12
7.in-addr.arpa. failed on the DNS server 192.58.128.30

            DNS server: 193.0.14.129 (k.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.12
7.in-addr.arpa. failed on the DNS server 193.0.14.129

            DNS server: 198.41.0.4 (a.root-servers.net.)
               1 test failure on this DNS server
               This is not a valid DNS server. PTR record query for the 1.0.0.12
7.in-addr.arpa. failed on the DNS server 198.41.0.4

         Summary of DNS test results:

                                            Auth Basc Forw Del  Dyn  RReg Ext
               ________________________________________________________________
            Domain: DOMAIN.com
               DC01                    PASS PASS FAIL PASS WARN PASS n/a

         ......................... DOMAIN.com failed test DNS

It looks pretty horrific - DNS is failing at a basic level! It turns out that the actual issue is an old version of DCDIAG.EXE. After several hours and a lot of head scratching I checked the versions of the DCDIAG.EXE (normally c:\Program Files\Support Tools\dcdiag.exe) and "Lo! And Behold!" the version was different. I downloaded the Windows Server 2003 Support Tools R2, uninstalled the old version (v5.2.3790.1800) and installed the new one (v5.2.3790.3959).

Et voila! The working test...

P:\>dcdiag /test:dns

Domain Controller Diagnosis

Performing initial setup:
   Done gathering initial info.

Doing initial required tests

   Testing server: SITE\DC01
      Starting test: Connectivity
         ......................... DC01 passed test Connectivity

Doing primary tests

   Testing server: SITE\DC01

DNS Tests are running and not hung. Please wait a few minutes...

   Running partition tests on : ForestDnsZones

   Running partition tests on : DomainDnsZones

   Running partition tests on : Schema

   Running partition tests on : Configuration

   Running partition tests on : DOMAIN

   Running enterprise tests on : DOMAIN.com
      Starting test: DNS
         Test results for domain controllers:

            DC: DC01.DOMAIN.COM
            Domain: DOMAIN.com

               TEST: Dynamic update (Dyn)
                  Warning: Dynamic update is enabled on the zone but not secure
DOMAIN.com.

         Summary of DNS test results:

                                            Auth Basc Forw Del  Dyn  RReg Ext
               ________________________________________________________________
            Domain: DOMAIN.com
               DC01                    PASS PASS PASS PASS WARN PASS n/a

         ......................... DOMAIN.com passed test DNS

Event Type: Error
Event Source: Userenv
Event Category: None
Event ID: 1030
Date:  10/09/2009
Time:  06:24:29
User:  NT AUTHORITY\SYSTEM
Computer: DC
Description:
Windows cannot query for the list of Group Policy objects. Check the event log for possible messages previously logged by the policy engine that describes the reason for this. For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

Event Type: Error
Event Source: Userenv
Event Category: None
Event ID: 1058
Date:  10/09/2009
Time:  06:24:29
User:  NT AUTHORITY\SYSTEM
Computer: DC
Description:
Windows cannot access the file gpt.ini for GPO CN={31B2F340-016D-11D2-945F-00C04FB984F9},CN=Policies,CN=System,DC=DOMAIN,DC=LCL. The file must be present at the location <\\DOMAIN.LCL\sysvol\DOMAIN.LCL\Policies\{31B2F340-016D-11D2-945F-00C04FB984F9}\gpt.ini>. (Windows cannot find the network path. Verify that the network path is correct and the destination computer is not busy or turned off. If Windows still cannot find the network path, contact your network administrator. ). Group Policy processing aborted. For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

On the affected machines, when navigating to \\DOMAIN.LCL there were no shares available, however navigating to \\DC shows the NETLOGON and SYSVOL shares. Pinging DOMAIN.LCL and then the DC showed that the IP addresses were not the same as expected, DOMAIN.LCL was resolving to the backup network, whereas DC was resolving to the servers LAN IP.

I checked the DNS records for the server, which were correct. Investigating the adaptor binding settings under Control Panel > Network Connections > Advanced > Advanced Settings showed that the backup network's adaptor was first in the list. I moved the adaptor for the LAN to the top of the list and OK'd my way out. I restarted the NETLOGON service and the issue was solved.

Windows servers have never been particularly good at being multi-homed, especially domain controllers. My advice comes from some bitter experience...

• If you have multiple network adaptors for extra bandwidth/redundancy/resiliance, then I would strongly recommend using Teamed adaptors, most of the major manufacturers' drivers and management software support it. This will eliminate any issues with multi-homing because as far as the server is concerned, it has one adaptor.
• If you have multiple network adaptors for different network segments and you're using RRAS to route between them, I would strongly suggest not using a Domain Controller at all for this purpose. Better yet, buy a hardware router.
• If you have multiple network adaptors for different purpose networks (e.g. a LAN, a backup network and an iSCSI network) then make sure you do the following:
  • Disable "File and Printer Sharing for Microsoft Networks" and "Client for Microsoft Networks" on all but the LAN adaptor.
  • Ensure that your LAN adaptor is the FIRST adaptor in the bindings in the advanced network settings.

 Hope that helps!