Some useful VMware related diagrams
vSphere 6 ESXTOP quick Overview for Troubleshooting
VMware vSphere 5 Memory Management and Monitoring diagram
Concepts and best practices in Resource Pools
Installing VMware Site Recovery Manager (SRM) 5.5 in the Lab
I’ve been playing about with a compact SRM install in my lab – since I have limited resources and only one site I wanted to create a run-through for anyone learning SRM to be able to do it in their own lab too. I am creating two sites on the same IP subnet (pretend it’s a stretched LAN across two sites) and will be protecting a single, tiny Linux web server using vSphere Replication. I’m aiming to cover SAN based replication in a later post.
Below is the list of hosts and VMs running for this exercise:
- ESXi-01 – my “Protected Site” – this is running DC-01, VC-01, SRM-01 and VRA-01 (to be installed later)
- ESXi-02 – my “Recovery Site” – this is running VC-02, SRM-02 and VRA-02 (to be installed later)
- DC-01 – this is my domain controller, I’m only going to use one DC for both “sites” as I don’t have the compute resource available to have a second running. This is also my Certificate Authority.
- VC-01 – this is my primary Virtual Center server, it’s a Windows 2012 R2 server. It is managing ESXi-01.
- VC-02 – this is my “recovery site” and it’s a Virtual Center Server Appliance (VCSA). It is managing ESXi-02
- SRM-01 – “protected site” SRM server, base install of Windows Server 2012 at this point
- SRM-02 – “recovery site” SRM server, base install of Windows Server 2012 at this point
- WEB-01 – this is a really, really, basic Ubuntu web server I’ve deployed from a template to use for testing.
Right – without further ado, let’s get stuck in!
VCSA 5.5 Web Client fails to log on with “SSL certificate verification failed”
This had me scratching my head, what seemed to be a common problem wasn’t fixed by the common solution. It was actually my fault – too familiar with the product and setting things up too quickly to test.
I installed a VCSA 5.5 instance in my lab as a secondary site for some testing and during the process found I couldn’t log on to the web client – it failed with the error:
Failed to connect to VMware Lookup Service https://vCVA_IP_address:7444/lookupservice/sdk – SSL certificate verification failed.
I had a closer look at the certificate being generated and noticed that the Subject Name was malformed “CN=vc-02.definit.loca” – that led me to the network config of the VCSA. I’d entered the FQDN into the “host name” field, which was in turn being passed to the certificate generation, truncated and throwing the SSL error. Changing the FQDN back to the host name “VC-02” and regenerating the certificate resolved the issue.
If you do have to follow that process, remember to disable the SSL certificate regeneration after it’s fixed – otherwise you’ll suffer slow boot times!
I’ll put that one down to over-familiarity with the product!
VMware vSphere Mobile Watchlist released
For those of you unaware VMware recently released the VMware vSphere Mobile Watchlist
What does it do?
“VMware vSphere Mobile Watchlist allows you to monitor the virtual machines you care about in your vSphere infrastructure remotely on your phone. Discover diagnostic information about any alerts on your VMs using VMware Knowledge Base Articles and the web. Remediate problems from your phone by using power operations or delegate the problem to someone on your team back at the datacenter.”
- REMEDIATE REMOTELY
Use power operations to remediate many situations remotely from your device.
- VMS AT A GLANCE
Review the status of these VMs from your device including: their state, health, console and related objects.
I have been using it for a day or so and I have found it very useful, presently I have it installed on my Android Phone and Tablet.
If you consider using this in conjunction with VPN or whatever your preferred secure method to connect to your work LAN when you are “out and about” its a great way to quickly take a look at any problematic VMs without needing to fire up your laptop.
Its available on Android and iOS and is well worth a quick look.
Avoid SSO Admin lockout – a.k.a. your first task after installing vSphere SSO
In my post yesterday (vexpert.me/hS) I talked about how to recover from an expired default SSO administrator password – this prompted a discussion on twitter with Anthony Spiteri (@anthonyspiteri) and Grant Orchard (@grantorchard) about the defaults for expiration and how to mitigate the risk.
The first solution is to modify the password expiration policy for SSO. I’m not advocating this necessarily – I think that expiring passwords ensure that you change them regularly and increase the overall security of your SSO solution. However, I can envisage situations (similar to mine) when the SSO administrator account is not used for a long time and expired – that causes headaches.
To modify the SSO password policy log onto the vSphere Web Client as the SSO admin (admin@system-domain for 5.1 or email@example.com 5.5) and select Administration, then Sign-On and Discovery > Configuration. Select the Policies tab – you should see the default config:
Click edit and set the password policy as required. This only applies to SSO users (i.e. those in the System-Domain or vSphere.local domains). To set the password to never expire in 5.1, set the Maximum Lifetime to 0 – for vSphere 5.5 you need to set to 9999 (Thanks to Hywel for his comments). IF you chose to do that, I’d beef up the complexity of your password policy to include upper, lower, numeric and special characters and increase the length from 8 to 13.
Similarly, you can edit the lockout policy which by default will lock you out if it has 3 failed attempts within 24 days. It will lock you out for 15 minutes. Setting the lockout time to 0 forces a manual unlock by an SSO admin.
The second option seems preferable to me (and Anthony and Grant) – that is to add some AD users or groups to the SSO administrators group. To do this, again log in as an SSO admin and select Administration, then Access > SSO Users and Groups, then the Groups tab. Select “__Administrators__” and click on the add principals button below. Select your AD domain from in the Identity Source field and search for your required user or group. Add them and click OK. Now those users, or group members have the ability to log on and reset or unlock the SSO admin account. AD accounts are obviously subject to your AD password policy, but can be reset independently of SSO and therefore don’t require you to use some command-line kung-fu to unlock.
vSphere Security: Understanding ESXi 5.x Lockdown Mode
This is the first article in a series of vSphere Security articles that I have planned. The majority of this article is based on vSphere/ESXi 5.1, though I will include any 5.5 information that I find relevant.
I think lockdown mode is a feature that is rarely understood, and even more rarely used. Researching this article I’ve already encountered several different definitions that weren’t quite right. As far as I can see there are no differences between lockdown more in 5.5 and 5.1.
The vSphere Security guide says (emphasis mine):
To increase the security of your ESXi hosts, you can put them in lockdown mode. In lockdown mode, all
operations must be performed through vCenter Server. Only the vpxuser user has authentication
permissions, no other users can perform operations against the host directly.
In short, lockdown mode means you can ONLY manage the host via vCenter. The only exception is via the DCUI. (more…)
Installing VMware vSphere Update Manager Download Service and publishing via IIS
The vSphere UMDS provides a way to download patches for VMware servers that have an air-gap, or for some reason aren’t allowed to go out to the internet themselves – in my case a security policy prevented a DMZ vCenter Server from connecting to the internet directly. The solution is to use UMDS to download the updates to a 2nd server that was hosted in the DMZ and then update the vCenter Server from there. It also can save on bandwidth if you’re running multiple vCenter Servers, which again was the case (though bandwidth isn’t really a constraint). (more…)
vCenter 5.1 Single Sign On Multi-Site error: User credentials are incorrect or empty
The error thrown was:
User credentials are incorrect or empty. Provide correct credentials.
After a couple of hours online with VMware support I took a guess at the problem. On the existing Single Sign On Configuration I have added the Active Directory domain DefinIT and in order to enable integrated authentication from the vSphere Client I moved it to the top of the list – this meant that System-Domain is no longer the default authentication domain. The SSO admin account (admin@System-Domain) is a part of that domain and so my guess is that the installer tries to authenticate using firstname.lastname@example.org rather than System-Domain, which of course failed.
Moving System-Domain back to the top of the list allowed me to install correctly, and once finished I could drop it back down to allow integrated authentication again.
Powershell – Generate Microsoft CA signed SSL certificates with vSphere 5.1
The process of requesting certificates for vSphere 5.1 is a fairly grim, manual process. It’s repetitive and easy to make a mistake on any step of the way. Since I’ve got to do this for quite a few VirtualCenter Servers, I thought I’d script the certificate generation if nothing else. I am following the excellent documentation provided in Implementing CA signed SSL certificates with vSphere 5.1 and more specifically in Creating certificate requests and certificates for vCenter Server 5.1 components.
The script assumes that:
- You have a working Certificate Authority
- You are in an Active Directory domain environment
- You have the relevant permissions to modify Certificate Templates, Request and Issue certificates.
- You have installed OpenSSL v1.0.1c or later.
You will need to modify the configuration section to suit your environment and the $WorkingDir folder should exist before you run the script. (more…)