VCAC 6.0 build-out to distributed model – Part 1: Certificates

This is the first article in a series about how to build-out a simple vCAC 6 installation to a distributed model.


Simple vCAC deployment

In a simple installation you have the Identity Appliance, the vCAC appliance (which includes a vPostgres DB and vCenter Orchestrator instance) and an IaaS server. The distributed model still has a single Identity Appliance but clusters 2 or more vCAC appliances behind a load balancer, backed by a separate vPostgres database appliance. The IaaS components are installed on 2 or more IaaS Windows servers and are load balanced, backed by an external MSSQL database. Additionally, the vCenter Orchestrator appliance is used in a failover cluster, backed by the external vPostgres database appliance.

The distributed model can improve availability, redundancy, disaster recovery and performance, however it is more complex to install and manage, and there are still single points of failure – e.g. the vPostgres database is not highly available and although protected by vSphere HA could be the cause of an outage. Clustering the database would provide an improved level of availability but may not be supported by VMware. Similarly the Identity Appliance is currently a single point of failure, although there are also options for high availability there too.

An overview of the steps required is below:

  • Issue and install certificates
  • Deploy an external vPostgres appliance and migrate the vCAC database
  • Configure load balancing
  • Deploy a second vCAC appliance and configure clustering
  • Install and configure additional IaaS server
  • Deploy vCenter Orchestrator Appliance cluster


vCenter 5.5 Certificate Toolkit for Distributed Environments

SSL PadlockIf you’ve had the dubious pleasure of generating and installing vCenter certificates, you’ll know that it’s not the greatest of fun. When VMware released the SSL Certificate Automation Tool, it helped hugely, especially when you use Derek Seaman’s excellent SSL toolkit. I know that there are hours and hours of work put into this script by Derek and I want to thank him for that – it’s a massive time saver. This modification is to fit a different set of circumstances – “standing on the shoulders of giants” – and should in no way be seen as me criticising or stealing Derek’s work.

This week, while using the SSL Certificate Automation Tool and Derek’s script, I encountered a couple of things I felt could be improved for a more complex environment.

  1. The script is not written to handle distributed setups – e.g. different vSphere components on different servers.
  2. The script will handle root and a single subordinate CA, but not a third level – this requires some manual fudging.
  3. The script still creates Java Keystore .jks files, .pfx and .p12 files and properties and ID files for the SSO – these are all no longer required for vSphere 5.5 with the SSL Certificate Automation tool.

Distributed Setups

I’ve modified the script to use an array of PSObjects for $WServices rather than listing the service names. This means I can provide an FQDN for each service as a property: these are used throughout the script to generate certificate requests for each service with the correct FQDN.

The function CreateCSR now uses the FQDN property of the $WServices array – and I have added a DNS lookup to add the IP address to the CSR automatically as an IP and DNS subjectAltName. Each generated CSR is specific to the FQDN provided at the start of the script.

Multiple Subordinate CAs

The environments I am working on have a fairly standard Microsoft Certificate Services PKI setup: at the top there’s a Root CA, under that there’s a Policy CA, and under that there are Issuing CAs.

I have modified the script to use an array variable $CAs, which contains a list of CA FQDNs. The function DownloadRoot cycles through those and attempts to download each CA’s certificate in turn. The certificates and saved as “CA64-x.cer”, where x is the number of the CA in order.

For example, the Root CA is first in the $CAs array, and is downloaded first. The file is saved as CA64-1.cer. The next CA in the list is my Policy CA, which is saved as CA64-2.cer. Finally, the Issuing CA is saved as CA64-3.cer.

The naming and ordering of the CAs and their certs is important because it’s important to get the chain correct in the .pem files used in the SSL Certificate Automation tool.

As with Derek’s original script, if you’re not able to access the CAs from the vCenter Server (or wherever you’re running the script) then you need to manually download the files and create the CA64-x.cer files and place them in the $Cert_Dir folder – they will be detected and used by the script.

Generating only required files

The only files required by the SSL Certificate Automation Tool are a .pem file containing the entire chain and a .key file containing the private key for the issued certificate.

To generate the .pem file, we need to copy the contents of the CA certificates from root to leaf, starting with the leaf certificate, then the issuing CA, any intermediate CA and finally the root CA. The image below shows the order the certificates need to be pasted into the file.

PEM Certificate Format

I’ve modified the CreatePEMFiles function to generate “RootChain.pem”, which is a concatenation of the root certificates in the correct order. It then cycles through the Services and copies the contents of the generated certificate file and “RootChain.pem” to create the .pem file required by the SSL Certificate Automation Tool.

I’ve removed the additional steps for the Java KeyStore (.jks) files, which were required for SSO 5.1 but aren’t actually needed for 5.5. Similarly, steps to create the .pfx and .p12 files are removed as they are no longer required.

Functionality I’ve removed

There are certain functions I’ve removed when it made no sense to keep them, for example generating certificates for the Linux appliance. This makes no sense as by definition they’ll be on the same server/FQDN.

The function VCFQDN has been removed since the FQDN of services is provided in the $WServices array.

The function DownloadSub has been removed since the DownloadRoot function has been modified to download all the CA certificates, including the Subordinate CAs.

The function WinVCCheck has been removed, this checked for the SSO install path and set up an alias to the keytool.exe installed there. These were used in functions that are no longer required.

The function CAHashes, which created the <hash>.0 files of the root certificates has been removed – again these are no longer required in 5.5.

The function CreateSSOFiles has also been removed, since the SSL Certificate Automation tool no longer requires these files to be manually generated.

Running the script

The script runs in the same way as Derek’s original – albeit with a few options removed. You need to edit the script before your first run to populate the details of your environment.

If you can’t access all the CA’s in your environment (e.g. offline root, or firewalled) then you will need to download your CA certificates as base64 encoded certs. Start at the top-most level – the root CA – and export it as CA64-1.cer. The next one should be CA64-2.cer and so on.

Other than that, the script runs as Derek’s. When it’s run, you will have a folder with the certificates in .pem format, and the matching private keys in a .key format. Copy the ssl-environment.bat in to replace the one in the folder for the SSL Certificate Automation Tool and run the tool to update your environment.


Download the script

The modified script is available to download here:

Generating and Installing CA Signed Certificates for VMware SRM 5.5

image I’m fairly new to SRM, but even so this one seemed like a real head-scratcher! If you happen to be using CA signed certificates on your “protected site” vCenter and “recovery site” vCenter servers, when you come to linking the two SRM sites you encounter SSLHandShake errors – basically SRM assumes you want to use certificates for authentication because you’re using signed certificates. If you use the default self-signed certificates, SRM will default to using password authentication (see SRM Authentication). Where the process fails is during the “configure connection” stage, if either one of your vCenter servers does not have CA signed and the other does (throws an error that they are using different authentication methods) or that you are using self-signed certificates for either SRM installation (throws an error that the certificate or CA could not be trusted).

SRM server ‘vc-02.definit.local’ cannot do a pair operation. The reason is: Local and remote servers are using different authentication methods.

image (more…)

Why secure your vSphere environment with valid SSL certificates?

vmware logoIt’s no secret that installing certificates from an internal CA is a pain in the…vCenter, but having just gone through the process of updating 3 vCenter installations with the 5-7 certificates required for each server I was asked “just why is it we need to do this again?”

Why does it require multiple certificates for my vCenter server?

In short, each service requires a certificate because it could feasibly be on a server (or servers) of it’s own – take this hypothetical design – each role is hosted on it’s own VM, and there are 7 certificates required – SSO, Inventory Service, vCenter Server, Orchestrator, Web Client, Log Browser and Update Manager. If you install all these services on one server you still have to create certificates for those individual services.

Certs - vCenterSeperate (more…)

Powershell – Generate Microsoft CA signed SSL certificates with vSphere 5.1

vmware logoThe process of requesting certificates for vSphere 5.1 is a fairly grim, manual process. It’s repetitive and easy to make a mistake on any step of the way. Since I’ve got to do this for quite a few VirtualCenter Servers, I thought I’d script the certificate generation if nothing else. I am following the excellent documentation provided in Implementing CA signed SSL certificates with vSphere 5.1 and more specifically in Creating certificate requests and certificates for vCenter Server 5.1 components.

The script assumes that:

  1. You have a working Certificate Authority
  2. You are in an Active Directory domain environment
  3. You have the relevant permissions to modify Certificate Templates, Request and Issue certificates.
  4. You have installed OpenSSL v1.0.1c or later.

You will need to modify the configuration section to suit your environment and the $WorkingDir folder should exist before you run the script. (more…)

Trouble with SCOM 2007 R2 Certificates? Validate the entire PKI path!

MSFT-System-Center-logoI learned something new today: SCOM 2007 R2 certificate based communications not only checks the validity of the certificate you use, but also the CA that issued it…let me expand:

Like many organisations there is a root CA (we’ll call it ROOTCA01), and then a subordinate CA (we’ll call that SUBCA01). OPSMGM01 has a certificate to identify itself and has certificates for ROOTCA01 and SUBCA01 in it’s Trusted Root Certificate Authorities.

The certificate to secure the connection between OpsMgr Gateway (OPSGW01) and the OpsMgr Management Server (OPSMGM01) is issued by SUBCA01 and is installed on OPSGW01, and to validate the certificate chain SUBCA01’s certificate is also installed in the Trusted Root Certification Authorities. Opening OPSGW01’s certificate and examining the Certificate Path tab shows the certificate is valid all the way up to the issuing CA – SUBCA01.

The connection will not work – OPSGW01 logs the following events:

Log Name:      Operations Manager
Source:        OpsMgr Connector
Date:          05/01/2012 10:18:28
Event ID:      21016
Level:         Error
Description:   OpsMgr was unable to set up a communications channel to and there are no failover hosts.  Communication will resume when is available and communication from this computer is allowed.

Log Name:      Operations Manager
Source:        OpsMgr Connector
Date:          05/01/2012 10:18:25
Event ID:      20070
Level:         Error
Description:   The OpsMgr Connector connected to, but the connection was closed immediately after authentication occurred.  The most likely cause of this error is that the agent is not authorized to communicate with the server, or the server has not received configuration.  Check the event log on the server for the presence of 20000 events, indicating that agents which are not approved are attempting to connect.

Log Name:      Operations Manager
Source:        OpsMgr Connector
Date:          05/01/2012 10:18:24
Event ID:      21002
Level:         Warning
Description:   The OpsMgr Connector could not accept a connection from because mutual authentication failed.

Log Name:      Operations Manager
Source:        OpsMgr Connector
Date:          05/01/2012 10:18:24
Event ID:      20067
Level:         Warning
Description:   A device at IP attempted to connect but the certificate presented by the device was invalid.  The connection from the device has been rejected.  The failure code on the certificate was 0x800B0109 (A certificate chain processed, but terminated in a root certificate which is not trusted by the trust provider.).

It’s the last event that led me to check the certificate chain for the SUBCA01 certificate, which was installed and trusted but did not validate up the chain to ROOTCA01. Installing the ROOTCA01 certificate resolved this issue.

Configuring a Guest wireless network with restricted access to Production VLANs

It’s a fairly common requirement – setting up a guest WiFi network that is secure from the rest of your LAN. You need a secure WLAN access for the domain laptops which has full access to the Server and Client VLANs, but you also need a guest WLAN for visitors to the office which only allows internet access. Since the budget is limited, this must all be accomplished via a single Access Point – for this article, the access point is a Cisco WAP4410N. (more…)

Configuring SSTP VPN connections to Threat Management Gateway 2010

TMG2010SSTP or SSL VPN connections are great for people working on client sites or behind very restrictive firewalls – they only require HTTPS (port 443) to be open to be able to connect. Unfortunately, you need to be running Windows 7 or Server 2008 (or newer) in order to make use of them. Threat Management Gateway 2010 is one option for an SSL VPN endpoint.

SSTP VPN Requirements

  • Clients must be Windows 7/Server 2008 or newer
  • Certificate – either commercial or an internal Certificate Authority
  • Published CRL – SSTP clients check for the Certificate Revocation List of the CA
  • If you already have an SSL listener (e.g. for Exchange publishing rules) then you need a dedicated IP address for the SSTP connection


Using System Center Operations Manager 2007 R2 Audit Collection Services for remote, DMZ or workgroup servers

MSFT-System-Center-logoSCOM 2007 R2’s Audit Collection Services (ACS from now on) is very useful for meeting compliance (e.g. Sarbanes Oxley) and security audit requirements – working with financial companies often requires such compliance. It’s pretty simple to install in a domain environment – you run the installer to create a collection server, then activate the forwarder on the client servers.

When it comes to servers you really want to audit, those that are by definition more at risk from security breach because they are publicly accessible, it’s not so straightforward. Take for example that web server, or FTP host in your DMZ, certainly not domain joined and probably bombarded by daily brute force password attacks. Select the SCOM agent in the console and enable Audit Collection Services?