As a vExpert, I am blessed to get 1000 CPU hours access to Ravello’s awesome platform and recently I’ve been playing with the AutoLab deployments tailored for Ravello.

If you’re unfamiliar with Ravello’s offering (where have you been?!) then it’s basically a custom hypervisor (HVX) running on either AWS or Google Cloud that allows you to run nested environments on those platforms. I did say it’s awesome.

As an avid home-lab enthusiast Ravello initially felt weird, but having used it for a while I can definitely see the potential to augment, and in some cases completely replace the home lab. I spent some time going through Nigel Poulton’s AWS course on Pluralsight to get a better understanding of the AWS platform and I think that helped, but it’s definitely not required to get started on Ravello.

One more thing to add before I start the setup – even if I didn’t have 1000 hours free, the pricing model means that you could run your lab on Ravello for a fraction of the cost of a higher spec home lab. It’s definitely an option to consider unless you’re running your lab 24/7.

(more…)

Note: This falls under the “I don’t think this is supported” category – use this method at your own peril!

As part of some testing I’ve been doing for vRealize Automation DR scenarios, I wanted to test changing the IP address of a HA PSC pair using a script (think SRM failover to a new subnet).

What I didn’t want to do was simply edit the connections directly – quite often with the VMware appliances there are scripts on start-up to ensure the configuration is correct and consistent – what I wanted was to be able to find a more supported and reliable way.

Fortunately the VAMI scripts are deployed on most appliances and are included on the PSC. I was able to work out a process (mostly by trial and error!) of getting the IP change to stick.

# Update the network IP address (this is for IPv4, there are options for IPv6 too, and DHCP)
/opt/vmware/share/vami/vami_set_network eth0 STATICV4 192.168.10.52 255.255.255.0 192.168.10.1
# This updates the IP in /etc/hosts - requires the FQDN as an argument or sets it to localhost.localdomain
/opt/vmware/share/vami/vami_set_hostname vra.definit.local
# This makes the changes “stick” on reboot
/opt/vmware/share/vami/vami_ensure_network_configuration eth0
reboot

I successfully used the the Guest Script Manager package from the VMware Center of Excellence to store and execute the script via vRealize Orchestrator, as well as using a bash script actually on the host. This worked during my testing to modify both the IP addresses in a PSC HA Cluster, and allowed (with some DNS changes) the fail-over to a completely different subnet.

I’m not sure how supported this is, but this process can recover a vSphere 6 vCenter Server Appliance or Platform Services Controller when you’ve lost the root password.

Download the OpenSUSE Rescue CD – http://download.opensuse.org/distribution/13.2/iso/

Mount the CD to the PSC Appliance

Reboot the appliance and enter the BIOS setup using F2, configure the CD-ROM as first-boot device. Save and exit to reboot into the SUSE Live-CD.

Once the Live-CD has booted to a desktop, you’ll see a 12GB volume at the top – that’s your PSC appliance root. Double click to open the disk and then copy the path, we’ll need it later.

Next open a shell console and change the root password (of the live CD root user) to something memorable

sudo passwd root

Next, open the /etc/shadow file and copy the root user’s password:

sudo cat /etc/shadow | grep “root”

Copy this line into the shadow file on the 12GB partition we looked at earlier and replace the existing “root” line.

sudo vi /run/media/linux/<GUID>/etc/shadow

Paste the updated root password into the file, replacing the old. use :wq! to force the file to write and quit. Reboot, remove the CD and boot the the appliance.

Log in to the console interface and reset the password correctly.

Providing a highly available single sign on for vRealize Automation is a fundamental part of ensuring the availability of the platform. Traditionally, (vCAC) vRA uses the Identity Appliance and relies on vSphere HA to provide the availability of the SSO platform, but in a fully distributed HA environment that’s not really good enough. It’s also possible to use the vSphere 5.5 SSO install in a HA configuration – however, many companies are making the move to the latest version of vSphere and don’t necessarily want to maintain a 5.5 HA SSO instance.

The vSphere 6 Platform Services Controller can be deployed as an appliance or installed on a Windows host – personally I am a huge fan of the appliances and I tend to use them in my designs because of the simplicity and ease of use. A pair of PSCs can be deployed as a highly available SSO solution for vRealize Automation 6.2, replacing the Identity Appliance or vSphere 5.5. SSO, using either a NetScaler or F5 load balancer to load balance connections and provide the availability.

Personally, I’d prefer to use an NSX Edge Services Gateway to load balance the PSCs, but at the time of writing the Edge does not support the “Ability to have session affinity to the same PSC node across all configured ports”. See KB2112736 for more details.

So, this guide will show you how to create a highly available pair of Platform Service Controllers, configure one as a subordinate Certificate Authority to a Microsoft Certificate Services CA, and then load balance them with a NetScaler VPX. Although I am using just two node, you can in fact use the same method to load balance up to four. (more…)

After deploying a new vSphere 6 vCenter Server Appliance (VCSA) and configuring the Platform Services Controller (PSC) to act as a subordinate Certificate Authority (CS), I was unable to register the NSX Manager to the Lookup Service. Try saying that fast after a pint or two!?

Attempting to register NSX to the Lookup Service would result in the following error:

NSX Management Service operation failed.( Initialization of Admin Registration Service Provider failed. Root Cause: Error occurred while registration of lookup service, com.vmware.vim.vmomi.core.exception.CertificateValidationException: Server certificate chain not verified )

Initially I thought that the NSX manager needed to somehow import the VMCA certificate to trust the Lookup Service certificate, however after reaching out to the NBSU ambassadors list I had a reply from Julienne Pham, a Technical Solutions Architect and CTO Ambassador with VMware Professional Services, who pointed me to the correct solution.

It seems that changing the PSC and vCenter certificates (even with the Certificate Manager tool) does not correctly update the service registration information. To quote VMware KB 2109074:

…the vCenter Server system uses a new certificate, but the service registration information on the Platform Services Controller is not updated

To resolve this issue, we need to use the ls_update_certs.py script to register the services correctly. (more…)

I tested vSphere 6 quite intensively when it was in beta, but I didn’t ever upgrade my lab – basically because I need a stable environment to work on and I wasn’t sure that I could maintain that with the beta.

Now 6 has been GA a while and I have a little bit of time, I have begun the lab upgrade process. You can see a bit more about my lab hardware over on my lab page.

Upgrading to VSAN 6.0

The upgrade process for VSAN 5.5 to 6.0 is fairly straight forward

• Upgrade vCenter Server
• Upgrade ESXi hosts
• Upgrade the on-disk format to the new VSAN FS

Other parts of this guide have covered the vCenter and ESXi upgrade, so this one will focus on the disk format upgrade. Once you’ve upgraded these you’ll get a warning on your VSAN cluster:

(more…)

I tested vSphere 6 quite intensively when it was in beta, but I didn’t ever upgrade my lab – basically because I need a stable environment to work on and I wasn’t sure that I could maintain that with the beta.

I will be upgrading

• vCenter Server Appliance – currently 5.5 update 1
• vSphere Update Manager – currently 5.5 update 1
• 3 HP N54L resource hosts
• 1 Intel NUC management host

In my lab I run various VMware software suites listed below, although I typically run them in nested environments to keep my lab install relatively clean.

• vCloud Director
• vRealize Automation
• vRealize Orchestrator
• NSX

Other considerations:

• VSAN – I currently run VSAN 5.5 and will need to upgrade to 6.0
• Update Manager – I’d prefer to update my hosts using Update Manager where possible
• Certificates – I currently use a Microsoft CA, I’d like to move to the VMCA as a subordinate CA
• Drivers – VMware changed the drivers supported in ESXi, some consumer grade drivers are blacklisted
• Backup – I use the excellent Veeam Backup and Replication to protect key lab machines, and I know that it doesn’t yet support vSphere 6. That’s a hit I can take in my lab.

To upgrade I need to first verify everything is compatible using the VMware Product Interoperability Matrixes.

High level plan

Having read a lot of vSphere 6 docs, my upgrade plan is as follows:

1. Upgrade vCenter Server Appliance
2. Upgrade vSphere Update Manager
3. Upgrade ESXi
4. Upgrade VSAN
5. Upgrade nested labs and other software suites

I tested vSphere 6 quite intensively when it was in beta, but I didn’t ever upgrade my lab – basically because I need a stable environment to work on and I wasn’t sure that I could maintain that with the beta.

Checking for driver compatibility

In vSphere 5.5, VMware dropped the drivers for quite a few consumer grade NICs – in 6 they’ve gone a step further and actually blocked quite a few of these using a VIB package. For more information, see this excellent article by Andreas Peetz.

To list the NIC drivers you’re using on your ESXi hosts, use the following command:

esxcli network nic list | awk ‘{print $1}’|grep [0-9]|while read a;do ethtool -i $a;done

As you can see from the results, my HP N54Ls are running 3 NICs, a Broadcom onboard and two Intel PCI NICs. Fortunately the Broadcom chip is supported and the e1000e driver I’m using is compatible with vSphere 6 and is in fact superseded by a native driver package. (more…)

I tested vSphere 6 quite intensively when it was in beta, but I didn’t ever upgrade my lab – basically because I need a stable environment to work on and I wasn’t sure that I could maintain that with the beta.

(more…)