Slow or failed logon to VCSA 5.5 with vCOps in the environment

vsphere logoRecently I encountered this problem in a customer site whereby the logon to VCSA 5.5 would either time out, or take 3-5 minutes to actually log on.

Running a netstat on the VCSA during the attempt to logon showed there was a SYN packet sent to the vCOps appliance on port 443 that never established a connection. Another check was attempting to connect using curl https://<vCOpsIP> –k  – this would time out.

Ensuring connectivity to the vCOps appliance over port 443 fixed the logon timeout issue – presumably a the connection attempt holds up the logon process (single threaded?!) which causes a timeout in the logon process.

vCenter 5.5 Certificate Toolkit for Distributed Environments

SSL PadlockIf you’ve had the dubious pleasure of generating and installing vCenter certificates, you’ll know that it’s not the greatest of fun. When VMware released the SSL Certificate Automation Tool, it helped hugely, especially when you use Derek Seaman’s excellent SSL toolkit. I know that there are hours and hours of work put into this script by Derek and I want to thank him for that – it’s a massive time saver. This modification is to fit a different set of circumstances – “standing on the shoulders of giants” – and should in no way be seen as me criticising or stealing Derek’s work.

This week, while using the SSL Certificate Automation Tool and Derek’s script, I encountered a couple of things I felt could be improved for a more complex environment.

  1. The script is not written to handle distributed setups – e.g. different vSphere components on different servers.
  2. The script will handle root and a single subordinate CA, but not a third level – this requires some manual fudging.
  3. The script still creates Java Keystore .jks files, .pfx and .p12 files and properties and ID files for the SSO – these are all no longer required for vSphere 5.5 with the SSL Certificate Automation tool.

Distributed Setups

I’ve modified the script to use an array of PSObjects for $WServices rather than listing the service names. This means I can provide an FQDN for each service as a property: these are used throughout the script to generate certificate requests for each service with the correct FQDN.

The function CreateCSR now uses the FQDN property of the $WServices array – and I have added a DNS lookup to add the IP address to the CSR automatically as an IP and DNS subjectAltName. Each generated CSR is specific to the FQDN provided at the start of the script.

Multiple Subordinate CAs

The environments I am working on have a fairly standard Microsoft Certificate Services PKI setup: at the top there’s a Root CA, under that there’s a Policy CA, and under that there are Issuing CAs.

I have modified the script to use an array variable $CAs, which contains a list of CA FQDNs. The function DownloadRoot cycles through those and attempts to download each CA’s certificate in turn. The certificates and saved as “CA64-x.cer”, where x is the number of the CA in order.

For example, the Root CA is first in the $CAs array, and is downloaded first. The file is saved as CA64-1.cer. The next CA in the list is my Policy CA, which is saved as CA64-2.cer. Finally, the Issuing CA is saved as CA64-3.cer.

The naming and ordering of the CAs and their certs is important because it’s important to get the chain correct in the .pem files used in the SSL Certificate Automation tool.

As with Derek’s original script, if you’re not able to access the CAs from the vCenter Server (or wherever you’re running the script) then you need to manually download the files and create the CA64-x.cer files and place them in the $Cert_Dir folder – they will be detected and used by the script.

Generating only required files

The only files required by the SSL Certificate Automation Tool are a .pem file containing the entire chain and a .key file containing the private key for the issued certificate.

To generate the .pem file, we need to copy the contents of the CA certificates from root to leaf, starting with the leaf certificate, then the issuing CA, any intermediate CA and finally the root CA. The image below shows the order the certificates need to be pasted into the file.

PEM Certificate Format

I’ve modified the CreatePEMFiles function to generate “RootChain.pem”, which is a concatenation of the root certificates in the correct order. It then cycles through the Services and copies the contents of the generated certificate file and “RootChain.pem” to create the .pem file required by the SSL Certificate Automation Tool.

I’ve removed the additional steps for the Java KeyStore (.jks) files, which were required for SSO 5.1 but aren’t actually needed for 5.5. Similarly, steps to create the .pfx and .p12 files are removed as they are no longer required.

Functionality I’ve removed

There are certain functions I’ve removed when it made no sense to keep them, for example generating certificates for the Linux appliance. This makes no sense as by definition they’ll be on the same server/FQDN.

The function VCFQDN has been removed since the FQDN of services is provided in the $WServices array.

The function DownloadSub has been removed since the DownloadRoot function has been modified to download all the CA certificates, including the Subordinate CAs.

The function WinVCCheck has been removed, this checked for the SSO install path and set up an alias to the keytool.exe installed there. These were used in functions that are no longer required.

The function CAHashes, which created the <hash>.0 files of the root certificates has been removed – again these are no longer required in 5.5.

The function CreateSSOFiles has also been removed, since the SSL Certificate Automation tool no longer requires these files to be manually generated.

Running the script

The script runs in the same way as Derek’s original – albeit with a few options removed. You need to edit the script before your first run to populate the details of your environment.

If you can’t access all the CA’s in your environment (e.g. offline root, or firewalled) then you will need to download your CA certificates as base64 encoded certs. Start at the top-most level – the root CA – and export it as CA64-1.cer. The next one should be CA64-2.cer and so on.

Other than that, the script runs as Derek’s. When it’s run, you will have a folder with the certificates in .pem format, and the matching private keys in a .key format. Copy the ssl-environment.bat in to replace the one in the folder for the SSL Certificate Automation Tool and run the tool to update your environment.


Download the script

The modified script is available to download here:

Some useful vSphere related diagrams

VMware.jpgOne of the things that never fails to amaze me are the superb PDF diagrams I occasionally stumble upon so i thought it would be a useful idea to list some of the the ones I have found on my travels.


VMware vSphere 5 Memory Management and Monitoring diagram








 Concepts and best practices in Resource Pools








[Read more…]

Extending a vCenter Orchestrator (vCO) Workflow with ForEach – Backing up all ESXi hosts in a Cluster

vCenter Orchestrator (vCO)In my previous post Backing up ESXi 5.5 host configurations with vCenter Orchestrator (vCO) – Workflow design walkthrough I showed how to create a workflow to back up host configurations, but it was limited to one host at a time. For this post I’m going to show how to create a new workflow that calls the previous one on multiple hosts using a ForEach loop to run it against each one. This was actually easier than I had anticipated (having read posts on previous versions of vCO that involved created the loops manually).

Ready? Here we go…

Create a new workflow – I’ve called mine “DefinIT-Backup-ESXi-Config-Cluster” and open the Schema view. Drag a “ForEach” element onto the workflow and the Chooser window pops up. This is a bit deceptive at first because it’s blank! However, if you enter some search text it will bring up existing workflows, so I searched for my “DefinIT-Backup-ESXi-Config” workflow that was created in that previous post. [Read more…]

Installing VMware Site Recovery Manager (SRM) 5.5 in the Lab

Site Recovery ManagerI’ve been playing about with a compact SRM install in my lab – since I have limited resources and only one site I wanted to create a run-through for anyone learning SRM to be able to do it in their own lab too. I am creating two sites on the same IP subnet (pretend it’s a stretched LAN across two sites) and will be protecting a single, tiny Linux web server using vSphere Replication. I’m aiming to cover SAN based replication in a later post.

Below is the list of hosts and VMs running for this exercise:

  • ESXi-01 – my “Protected Site” – this is running DC-01, VC-01, SRM-01 and VRA-01 (to be installed later)
  • ESXi-02 – my “Recovery Site” – this is running VC-02, SRM-02 and VRA-02 (to be installed later)
  • DC-01 – this is my domain controller, I’m only going to use one DC for both “sites” as I don’t have the compute resource available to have a second running. This is also my Certificate Authority.
  • VC-01 – this is my primary Virtual Center server, it’s a Windows 2012 R2 server. It is managing ESXi-01.
  • VC-02 – this is my “recovery site” and it’s a Virtual Center Server Appliance (VCSA). It is managing ESXi-02
  • SRM-01 – “protected site” SRM server, base install of Windows Server 2012 at this point
  • SRM-02 – “recovery site” SRM server, base install of Windows Server 2012 at this point
  • WEB-01 – this is a really, really, basic Ubuntu web server I’ve deployed from a template to use for testing.

Right – without further ado, let’s get stuck in!

[Read more…]