Unable to connect NSX to Lookup Service when using a vSphere 6 subordinate certificate authority (VMCA)

After deploying a new vSphere 6 vCenter Server Appliance (VCSA) and configuring the Platform Services Controller (PSC) to act as a subordinate Certificate Authority (CS), I was unable to register the NSX Manager to the Lookup Service. Try saying that fast after a pint or two!?

Attempting to register NSX to the Lookup Service would result in the following error:

NSX Management Service operation failed.( Initialization of Admin Registration Service Provider failed. Root Cause: Error occurred while registration of lookup service, com.vmware.vim.vmomi.core.exception.CertificateValidationException: Server certificate chain not verified )


Initially I thought that the NSX manager needed to somehow import the VMCA certificate to trust the Lookup Service certificate, however after reaching out to the NBSU ambassadors list I had a reply from Julienne Pham, a Technical Solutions Architect and CTO Ambassador with VMware Professional Services, who pointed me to the correct solution.

It seems that changing the PSC and vCenter certificates (even with the Certificate Manager tool) does not correctly update the service registration information. To quote VMware KB 2109074:

…the vCenter Server system uses a new certificate, but the service registration information on the Platform Services Controller is not updated

To resolve this issue, we need to use the ls_update_certs.py script to register the services correctly. [Read more…]

My #VCDX-CMA defense experience

Yesterday, I received the dreaded email

We regret to inform you that your attempt to achieve VCDX certification on June 09-11, 2015 in Frimley, UK was unsuccessful.

It wasn’t entirely unexpected, but somehow I still hoped my assessment of the defense was pessimistic and so it was nonetheless disappointing. It’s a big hit to not achieve something I have been focusing on for months and it is hard not to feel embarrassed that I didn’t make the grade. I am looking forward to receiving some feedback from the panelists and will be gearing up for another attempt in October.

Attaining VCDX is a step on a long learning journey and from the start I approached this defense as that – a learning experience. It’s an opportunity to focus and set a goal, and to push myself to reach that next step.

Design Defense

I felt that I defended my design relatively well – although the panelists exposed some issues, I felt that I was able to present explanations and reasoning behind each issue and I had anticipated a lot of the questions they did pose. Going in I felt that the defence of my design was something I should be all over, it was written from start to finish by myself. I may be wrong, and if the feedback is that I didn’t defend it well then I’ll need to reassess this.

What will I do differently?

  • Plug the gaps – where the panelists exposed an issue in my design I will re-work the solution to close that gap. Where something needed explanation I’ll expand it.
  • Not “over answer” questions – one of the things I learned from doing mocks was that I wasted time explaining things that I didn’t need to. I’ve seen advice to have a high, medium and low level explanation prepared for everything, and I think that’s a good approach. Give the minimum explanation unless pressed for more.

Design Scenario

I had not anticipated how nerves in the design scenario would affect me. The design methodology is something I do day in, day out – it’s not something that would normally cause me to stumble. What actually happened was that I panicked trying to do a whole detailed design in 45 minutes and lost the structure and method that I normally use. Normally, I would have a couple of weeks to work through the design process. Instead, I started to pull out the Business Requirements and then jumped ahead to a conceptual diagram and then moved back to start the use cases and then jumped to a physical design. Not following through the design method stopped me from moving through the design in a logical manner and I think that’s where I came undone.

What will I do differently?

  • Follow the methodology – clearly the panelists are not expecting me to produce a low level design in 45 minutes, they want to see me work through the method. They can’t do that if I don’t! Not following the method led to a random jumping around, no structure, and I doubt showed the panelists the confidence required.
  • Don’t rush in – stop, breathe, think – not for too long, but I started talking and ran down random avenues because I didn’t take a beat and calm myself.
  • Mock scenarios – I didn’t spend as much time doing mock scenarios as I did preparing to defend my design, certainly not with the time and pressure of the real defense. I’ll be getting as many mock scenarios together as I can.

Troubleshooting Scenario

Troubleshooting isn’t something that you end up doing as often when you’re working as a consultant/architect but it is something I do a lot in my lab. Having panicked in the previous scenario I tried to recover my composure somewhat for this one, but once again I think I rushed in too quickly. The scenario is not about the solution, it’s about the method – a structured, logical approach is key. I found it hard to judge how I did in this one, I did rush headlong down one particular avenue on more of a “gut feeling” than I should have, but we shall see what the feedback is.

What will I do differently?

  • Go slower – establish the facts and the basics before focusing in too quickly on what I think the problem is. The CMA troubleshooting scenario is 30 minutes long and after 20 minutes I had run out of ideas in the narrow field I had focused on.
  • Mock scenarios – again, the focus of my mocks was on the design defense and not the scenarios, something I should have prepared better for – with the timer.

All of these things are my reflections on the defense, and why I failed. I could quite easily be wrong and have completely missed the reasons I failed – hopefully the feedback will confirm my thoughts and help me re-focus on the right areas. It has been a huge learning experience, from getting my design ready to submit, preparing to defend it, and then defending. I will continue to learn and develop and push myself for this next milestone, and beyond.

People say it’s a huge achievement just to be invited to defend, and I don’t disagree, but I am more determined than ever not just to get the invite, but to successfully defend VCDX.

Automate the deployment of vROps (supporting vDS)

powercli-logoRecently I have been looking at William Lam‘s excellent post on automating the deployment of vROps.

After having a play around with it, to suit my own needs, I made some modifications to the Powershell script so it would support distributed switches.

[Read more…]

vSphere 6 Lab Upgrade – vCenter Orchestrator to vRealize Orchestrator

vsphere logoI tested vSphere 6 quite intensively when it was in beta, but I didn’t ever upgrade my lab – basically because I need a stable environment to work on and I wasn’t sure that I could maintain that with the beta.

Now 6 has been GA a while and I have a little bit of time, I have begun the lab upgrade process. You can see a bit more about my lab hardware over on my lab page.

Upgrading the vCenter Orchestrator Appliance

Upgrading the vCenter Orchestrator Appliance is child’s play – just log onto the admin interface at https://vco.fqdn.com:5480 using the root credentials.

Select the update tab, then click “Check Updates”. You should see appliance version 6.0.1 available, then click Install Updates [Read more…]

vSphere 6 Lab Upgrade – VSAN

vsphere logoI tested vSphere 6 quite intensively when it was in beta, but I didn’t ever upgrade my lab – basically because I need a stable environment to work on and I wasn’t sure that I could maintain that with the beta.

Now 6 has been GA a while and I have a little bit of time, I have begun the lab upgrade process. You can see a bit more about my lab hardware over on my lab page.

Upgrading to VSAN 6.0

The upgrade process for VSAN 5.5 to 6.0 is fairly straight forward

  • Upgrade vCenter Server
  • Upgrade ESXi hosts
  • Upgrade the on-disk format to the new VSAN FS

Other parts of this guide have covered the vCenter and ESXi upgrade, so this one will focus on the disk format upgrade. Once you’ve upgraded these you’ll get a warning on your VSAN cluster:


[Read more…]