With the release of vCenter Log Insight Public Beta (http://communities.vmware.com/community/vmtn/vcenter/vcenter-log-insight) I thought I’d strike while the iron is hot and run through the installation and configuration.
Deploying the OVF
This is such a bread and butter task that it doesn’t require more than a few words – it’s definitely worth looking at the Sizing PDF before you deploy (VMware-vCenter-Log-Insight-1.0-Beta-Virtual-Appliance-Sizing.pdf) as it’s not small even for a test installation. If you’re using less than the recommended 8GB RAM there are additional steps to change the heap size for performance.
The vSphere UMDS provides a way to download patches for VMware servers that have an air-gap, or for some reason aren’t allowed to go out to the internet themselves – in my case a security policy prevented a DMZ vCenter Server from connecting to the internet directly. The solution is to use UMDS to download the updates to a 2nd server that was hosted in the DMZ and then update the vCenter Server from there. It also can save on bandwidth if you’re running multiple vCenter Servers, which again was the case (though bandwidth isn’t really a constraint).
Installing UMDS is a really simple affair – run the <installer location>\umds\VMware-UMDS.exe installer and click through the options. The simplest option is to run the in-build SQL 2008 R2 Express database for the metadata, though you can create a 32-bit DSN and use an existing DB server if required. I set the patch download location to a D:\ that was attached to this VM and finished the install:
It takes a little while to install largely because of the SQL Express installation.
Configuring UMDS and downloading the patches
Once you’ve got it installed, it’s a bit of a tumbleweed moment…what next?! There’s no application or GUI installed – you need to read the documentation (or this blog ) to see what’s next!
If you’ve installed it in the default location, run a command prompt and navigate to: c:\Program Files (x86)\VMware\Infrastructure\Update Manager
If you run the “vmware-umds.exe –H” command, you’ll see the the full help output - I think the examples are the easiest way to work out what you want to do!
To add a new ESX 4.x/5.x patch depot URL
vmware-umds.exe -S --add-url https://hostname/index.xml --url-type HOST
To remove a URL
vmware-umds.exe -S --remove-url https://hostname/index.xml
To list all supported platforms for downloading ESX host updates
To enable downloading of ESX host and virtual appliance updates
vmware-umds.exe -S --enable-host --enable-va
To enable downloading of only ESX 4.0 and ESXi 4.0 host updates
vmware-umds.exe -S --disable-host
vmware-umds.exe -S -e esx-4.0.0 embeddedEsx-4.0.0
To disable downloading of only ESX 4.0 and ESXi 4.0 host updates
vmware-umds.exe -S --enable-host
vmware-umds.exe -S -e esx-4.0.0 embeddedEsx-4.0.0
To download updates based on the current configuration
To export all downloaded updates to F:\UMDS-store
vmware-umds.exe -S --default-export-store F:\UMDS-store
vmware-umds.exe -E --export-store F:\UMDS-store
Since I don’t want to download any unnecessary patches, I want to exclude ESX from the mix. To view the available platforms you can download patches for, use the –list-host-platforms argument:
The easiest way to do this is to enable all host downloads and disable the ESX downloads:
vmware-umds –S –-enable-host
vmware-umds –S -d esx-4.1.0 esx-4.0.0
Now I can trigger the initial download to get the patches down – this is a pretty verbose output, if you want to log it you can pipe to a text file. This will also take a while to download!
Since I am going to publish these updates using IIS for my vSphere Update Manager servers, I want to configure an export directory using the –E option – depending on your disk speed, this could take a while. It’s worth noting that this will double the size of your repository. I would prefer to be able to use the download location as the repository.
vmware-umds –E –export-store D:\UMDS\Export
You can verify your configuration using the –G option.
Configuring IIS to publish the UMDS Export
Assuming you already have IIS installed (or are capable of managing this feat) you can go ahead and configure your IIS to publish the export folder.
Configure a MIME type for .sig and .vib to use “application/octet-stream”, and check that .xml is “text/xml”.
Configure upstream VMware Update Manger Servers
Configuring the upstream servers is as simple as providing the URL of the site you’ve just published under the Update Manager Administration Console in the vSphere Client. Go to the Configuration tab, click Download Settings and then select “Use a shared repository”. Enter the URL and click “Validate URL”. Then click “Apply”. If you want to do the initial sync, click “Download Now”, if not wait for the scheduled download.
For extra credit…
- Install an SSL certificate to secure your IIS server
- Schedule the download of patches using Task Scheduler
- Schedule the export of patches to your IIS site using Task Scheduler
If you work in company with strict password compliance rules, for example under SOX, you might well have to change administrator passwords every month. Doing this on any more than a few hosts is tedious work – even on two hosts it seems like a waste of time logging on the host via SSH (or even enabling it first) before changing the password. Then we also need to audit the change, there’s no point making it for compliance reasons if we can’t then prove we did it!
I am absolutely thrilled to announce that I've been awarded vExpert 2013 - it's such an honour to be listed among these others and hopefully I can continue to contribute throughout the year. I am looking forward to getting stuck in to the vExpert programme.
The vExpert announcement is here: http://blogs.vmware.com/vmtn/2013/05/vexpert-2013-awardees-announced.html
In other news, one of DefinIT's contributing authors @SimonEady is a finalist for the VMware V.I.T. Competition and needs your votes!y I can continue making a contribution to the VMware community for the coming year. I'm looking forward to getting stuck in to the programme!
Please head on over to the voting page and vote for Simon!
It's no secret that installing certificates from an internal CA is a pain in the...vCenter, but having just gone through the process of updating 3 vCenter installations with the 5-7 certificates required for each server I was asked "just why is it we need to do this again?"
Why does it require multiple certificates for my vCenter server?
In short, each service requires a certificate because it could feasibly be on a server (or servers) of it's own - take this hypothetical design - each role is hosted on it's own VM, and there are 7 certificates required - SSO, Inventory Service, vCenter Server, Orchestrator, Web Client, Log Browser and Update Manager. If you install all these services on one server you still have to create certificates for those individual services.
A problem reared it's head over the weekend with one of our hosts' Fibre Channel HBAs negotiating it's way down to 2GB, and consequently introducing massive latency for the LUNs behind it. Analysis showed that the drivers for the HBA were over a year out of date so the suggested fix from VMware was to update the drivers. This is fine to do manually for a few hosts, but would be a real pain for the 300+ hosts in the environment I manage.
Enter VMware Update Manager - we already use it to patch and upgrade our hosts, but it can also apply host extensions - driver packages as well.
So then! of late my attention has been drawn to Cloud Credibility which is a fantastic place to help validate your own and others cloud expertise by completing various tasks.
Among other things it encourages you to read up on white papers, carry out lab work (Hands-on-labs), watch training and informational materials and thus rewarding you with points for you and your team. What is also great is points really do mean prizes!
I have recently become apart of a Team (Team - DefinIT) with the following well known Virtualisation bloggers and vExperts.
Barry Coombs - Virtualised Reality
Michael Poore - vSpecialist
Sam McGeown - DefinIT
This presents another great aspect to Cloud Credibility as it encourages team work with tasks and social/technical interactions.
If you haven't signed up I would strongly recommend you do so!
As some of you read previously, I had been experiencing disk latency issues on our SAN and tried many initial methods to troubleshoot and understand the root cause. Due to other more pressing issues this was placed aside until we started to experience VMs being occasionaly restarted by vSphere HA as the lock had been lost on a given VMDK file. (NOT GOOD!!)
3x vSphere 5.1 Hosts
2x 4port Nics 1GBe (allowing 2x iSCSi vmkernel ports per host for redundancy)
Dedicated Switching (isolated from the LAN) for iSCSi and vMotion (on seperate respective VLANs)
MSA2312i SAN G2 (with 4 Shelves)
The iSCSi Multipathing policy was set to Round Robin.
SIOC is enabled.
After a great deal of digging I resolved to contacting VMware support whom pointed me in turn to the SAN as the Host log files had the following..
<span style="color: #ff0000;">2013-02-20T14:35:38.026Z cpu8:51055)ScsiDeviceIO: 2316: Cmd(0x4124401f31c0) 0x1a, CmdSN 0x6796 from world 0 to dev "naa.600508e00000000078c4a59a76937603" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0. 2013-02-20T14:35:38.030Z cpu8:51055)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x85 (0x4124401f31c0, 91852) to dev "naa.600508e00000000078c4a59a76937603" on path "vmhba1:C1:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE 2013-02-20T14:35:38.030Z cpu8:51055)ScsiDeviceIO: 2316: Cmd(0x4124401f31c0) 0x85, CmdSN 0xab from world 91852 to dev "naa.600508e00000000078c4a59a76937603" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. 2013-02-20T14:35:38.030Z cpu8:51055)ScsiDeviceIO: 2316: Cmd(0x4124401f31c0) 0x4d, CmdSN 0xac from world 91852 to dev "naa.600508e00000000078c4a59a76937603" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. 2013-02-20T14:35:38.030Z cpu8:51055)ScsiDeviceIO: 2316: Cmd(0x4124401f31c0) 0x1a, CmdSN 0xad from world 91852 to dev "naa.600508e00000000078c4a59a76937603" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.</span>
So duely armed I contacted HP support whom immediately escalated the issue internally. During this time I had a very helpful conversation with a good friend
@VirtualisedReal whom pointed me in the direction of the HP MSA best practice document. I applied the subnetting configuration it suggested, which seperates the iSCSi ports A1 & B1 from A2 & B2 on seperate subnets and also configured each of the hosts 2 iSCSi vmkernel ports to point to the seperate paired iSCSi ports on the SAN.
When HP did eventually come back to me they suggested the SAN was perfectly fine, However! enough time had passed since the iSCSi port configuration change that I could already see a noticable drop in latency.
I waited another week (and since then) and I am very glad to say the latency is considerably lower with no reoccurance of the locks being lost on VM vmdk files.
PowerCLI Script to set RDM LUNs to Perennially Reserved – Fixes Slow Boot of ESXi 5.1 with MSCS RDMs
I've previously posted around this topic as part of another problem but having had to figure out the process again I think it's worth re-posting a proper script for this. VMware KB 1016106 is snappily titled "ESXi/ESX hosts with visibility to RDM LUNs being used by MSCS nodes with RDMs may take a long time to boot or during LUN rescan" and describes the situation where booting ESXi (5.1 in my case) takes a huge amount of time to boot because it's attempting to gain a SCSI reservation on an RDM disk used by MS Clustering Services. It also details the fix.
The process is fairly simple, but a bit labour intensive if you're doing it manually on a large cluster.
- Retrieve the ScsiCanonicalName for each RDM
- Set the configuration for each RDM on each Host to "PerenniallyReserved"