Fixing a couple of search/inventory service related errors with vCenter appliance 5.5 update 1c
So recently I came across an error in the vSphere windows “fat client” when trying to use the search field.
So a quick look at the VMware knowledge base brought up the following article
So I went ahead and followed the KB artricle and then tried to search again.. the following error was generated.
Also while logging into the vSphere web client the following error appears.
I had access to the SSO components etc.. but vCenter and related objects were null, now I have seen this issue before when the in use domain account has not been added to the vCenter as an admin so I re-logged with the firstname.lastname@example.org account (which had no errors when logging in) and browsed through to the vCenter server permissions tab and I could see all of the appropriate accounts listed with the correct permissions.
Slow or failed logon to VCSA 5.5 with vCOps in the environment
Recently I encountered this problem in a customer site whereby the logon to VCSA 5.5 would either time out, or take 3-5 minutes to actually log on.
Running a netstat on the VCSA during the attempt to logon showed there was a SYN packet sent to the vCOps appliance on port 443 that never established a connection. Another check was attempting to connect using curl https://<vCOpsIP> –k – this would time out.
Ensuring connectivity to the vCOps appliance over port 443 fixed the logon timeout issue – presumably a the connection attempt holds up the logon process (single threaded?!) which causes a timeout in the logon process.
Book review: Networking for VMware Administrators
I recently got my hands on a copy* of Chris Wahl and Steve Pantol’s Networking for VMware Administrators and was very keen to read it – especially given the reputation of the authors. I came to the book as someone who is at CCNA level (although now expired) and someone who regularly designs complex VMware networks using standard and distributed switches. I would class myself as having a fairly decent understanding of networking, though not a networking specialist.
The book starts out at from a really basic level explaining OSI, what a protocol is etc. and builds on the foundation set out as it progresses. Part I of the book gives are really good explanation of not only the basics of networking, but a lot of the “why” as well. If you’ve done CCNA level networking exams then you will know most of this stuff – but it’s always good to refresh, and maybe cover any gaps.
Part II of the book translates the foundations set out in Part I into the virtual world and takes you through the similarities and differences with between virtual and physical. It gives a good overview of the vSphere Standard Switch (VSS) and vSphere Distributed Switch (vDS) and even has a chapter on the Cisco 1000v. One of the really useful parts of the book are the lab examples and designs, which takes you though the design process and considerations to get to the solution. (more…)
Extending a vCenter Orchestrator (vCO) Workflow with ForEach – Backing up all ESXi hosts in a Cluster
In my previous post Backing up ESXi 5.5 host configurations with vCenter Orchestrator (vCO) – Workflow design walkthrough I showed how to create a workflow to back up host configurations, but it was limited to one host at a time. For this post I’m going to show how to create a new workflow that calls the previous one on multiple hosts using a ForEach loop to run it against each one. This was actually easier than I had anticipated (having read posts on previous versions of vCO that involved created the loops manually).
Ready? Here we go…
Create a new workflow – I’ve called mine “DefinIT-Backup-ESXi-Config-Cluster” and open the Schema view. Drag a “ForEach” element onto the workflow and the Chooser window pops up. This is a bit deceptive at first because it’s blank! However, if you enter some search text it will bring up existing workflows, so I searched for my “DefinIT-Backup-ESXi-Config” workflow that was created in that previous post. (more…)
What about VSAN for SMB/SME?
Unless you have been sleeping under a rock you will be aware that VSAN was launched last week and has gone GA today and from what I have seen so far I do think VSAN is a great product and I think VMware have done a superb job with it.
Aside from the -many- discussions on twitter and other channels regarding the then lack of licensing information and pricing I was eager to see if VMware would offer a “foundation” VSAN option for SMB/SME
Well with the Announcements I have seen today this would seem not to be the case.
VSAN requires a minimum of 3 hosts, so lets assume for arguments sake they will have 2 CPUs each, this will mean a VSAN install will set you back around a cool £9000 (happy to be corrected if this figure is now inaccurate) before you even go ahead and procure any disks etc. That is not exactly compelling me to rush out and buy it especially given a SAN in this area of the marketplace is not far above that price including disks.
So where is the SME/SMB love VMware? I am sure they are keenly aware that 50-60% of their business is SME/SMB (in the UK at least) so I am very disappointed to see little or no offerings for this marketplace.
Perhaps this comes down to their interpretation of an SME/SMB everyone I have so far spoken to has differing opinions so perhaps this should not be a surprise?
Perhaps they had no intention to pitch it at that marketplace?
Either way I will hold out hope that a “Foundation” option appears and SME/SMB businesses alike can share in the VSAN goodness.
If you work in an SME/SMB please let me know what you think. I am personally very keen for more opinions on the subject.
How to backup and restore your ESXi host config
There are many ways to tackle the problem of quickly redeploying or recovering ESXi hosts, Host profiles, Auto deploy etc.. however such options are either out of reach for SME/SMB users where their license does not cover such features or they have very small clusters of which Auto deploy etc would perhaps be considered overkill.
So how can we backup the config of our ESXi hosts? There is a great command you can use in vSphere CLI “vicfg-cfgbackup.pl“, which when used with certain switches can either back up or restore your ESXi host config.
Backing up a host
Quite simply you fire up your vSphere CLI client and run the command as shown below, make sure you define a file name as well as the destination folder or it will error.
You will then be prompted for authentication to the host, assuming you input the correct credentials the firmware configuration will be saved successfully to the folder you specified.
You may notice on my example I saved the file type as .tgz, you can drill into the .tgz file and see all of the config this process saves which is kind of handy if you want to be doubly sure it did the job correctly.
Restoring a host
So now you want to restore a host from a backup you have taken, we can use the same command but with the -l switch.
Important things to note
- This action will reboot your host
- This command will want to place your host in maintenance mode so therefore you will need to evacuate any VMs on the host.
- Placing the host into maintenance mode prior to running the command will not work and it will error, the process needs to place the host in maintenance mode itself.
- If you are running a small cluster you will likely need to disable HA while you perform this action to avoid errors being generated due to the lack of available resources.
Example error below
Successful restore below
I have found this to be really handy if I wish to restore a host to a previous running config, and by example will save you having to re-enter all of your network config etc.
VCSA 5.5 Web Client fails to log on with “SSL certificate verification failed”
This had me scratching my head, what seemed to be a common problem wasn’t fixed by the common solution. It was actually my fault – too familiar with the product and setting things up too quickly to test.
I installed a VCSA 5.5 instance in my lab as a secondary site for some testing and during the process found I couldn’t log on to the web client – it failed with the error:
Failed to connect to VMware Lookup Service https://vCVA_IP_address:7444/lookupservice/sdk – SSL certificate verification failed.
I had a closer look at the certificate being generated and noticed that the Subject Name was malformed “CN=vc-02.definit.loca” – that led me to the network config of the VCSA. I’d entered the FQDN into the “host name” field, which was in turn being passed to the certificate generation, truncated and throwing the SSL error. Changing the FQDN back to the host name “VC-02” and regenerating the certificate resolved the issue.
If you do have to follow that process, remember to disable the SSL certificate regeneration after it’s fixed – otherwise you’ll suffer slow boot times!
I’ll put that one down to over-familiarity with the product!
Reclaiming an SSD device from ESXi 5.5 Virtual Flash
After having a play with Virtual Flash and Host Caching on one of my lab hosts I wanted to re-use the SSD drive, but couldn’t seem to get vFlash to release the drive. I disabled flash usage on all VMs and disabled the Host Cache, then went to the Virtual Flash Resource Management page to click the “Remove All” button. That failed with errors:
“Host’s virtual flash resource is inaccessible.”
“The object or item referred to could not be found.”
In order to reclaim the SSD you need to erase the proprietary vFlash File System partition using some command line kung fu. SSH into your host and list the disks:
You’ll see something similar to this:
You can see the disk ID “t10.ATA_____M42DCT032M4SSD3__________________________00000000121903600F1F” and below it appended with the “:1” which is partition 1 on the disk. This is the partition that I need to delete. I then use partedUtil to delete the partition I just identified using the format below:
partedutil delete “/vmfs/devices/disks/<disk ID>” <partition number>
partedutil delete “/vmfs/devices/disks/t10.ATA_____M42DCT032M4SSD3__________________________00000000121903600F1F” 1
There’s no output after the command:
Now I can go and reclaim the SSD as a VMFS volume as required:
Hope that helps!
Avoid SSO Admin lockout – a.k.a. your first task after installing vSphere SSO
In my post yesterday (vexpert.me/hS) I talked about how to recover from an expired default SSO administrator password – this prompted a discussion on twitter with Anthony Spiteri (@anthonyspiteri) and Grant Orchard (@grantorchard) about the defaults for expiration and how to mitigate the risk.
The first solution is to modify the password expiration policy for SSO. I’m not advocating this necessarily – I think that expiring passwords ensure that you change them regularly and increase the overall security of your SSO solution. However, I can envisage situations (similar to mine) when the SSO administrator account is not used for a long time and expired – that causes headaches.
To modify the SSO password policy log onto the vSphere Web Client as the SSO admin (admin@system-domain for 5.1 or email@example.com 5.5) and select Administration, then Sign-On and Discovery > Configuration. Select the Policies tab – you should see the default config:
Click edit and set the password policy as required. This only applies to SSO users (i.e. those in the System-Domain or vSphere.local domains). To set the password to never expire in 5.1, set the Maximum Lifetime to 0 – for vSphere 5.5 you need to set to 9999 (Thanks to Hywel for his comments). IF you chose to do that, I’d beef up the complexity of your password policy to include upper, lower, numeric and special characters and increase the length from 8 to 13.
Similarly, you can edit the lockout policy which by default will lock you out if it has 3 failed attempts within 24 days. It will lock you out for 15 minutes. Setting the lockout time to 0 forces a manual unlock by an SSO admin.
The second option seems preferable to me (and Anthony and Grant) – that is to add some AD users or groups to the SSO administrators group. To do this, again log in as an SSO admin and select Administration, then Access > SSO Users and Groups, then the Groups tab. Select “__Administrators__” and click on the add principals button below. Select your AD domain from in the Identity Source field and search for your required user or group. Add them and click OK. Now those users, or group members have the ability to log on and reset or unlock the SSO admin account. AD accounts are obviously subject to your AD password policy, but can be reset independently of SSO and therefore don’t require you to use some command-line kung-fu to unlock.