Upgrading PKS with NSX-T from 1.0.x to 1.1

Written by Sam McGeown
Published on 29/6/2018 - Read in about 6 min (1220 words)

PKSYesterday, Pivotal Container Service 1.1 dropped and, as it’s something I’ve been actively learning in my lab, I wanted to jump on the upgrade straight away. PKS with NSX-T is a really hot topic right now and I think it’s going to be a big part of the future CNA landscape.

My Lab PKS 1.0.4 deployment is configured as a “NO-NAT with Logical Switch (NSX-T) Topology” as depicted in the diagram below (from the PKS documentation). My setup has these network characteristics:

  • PKS control plane (Ops Manager, BOSH Director, and PKS VM) components are using routable IP addresses.
  • Kubernetes cluster master and worker nodes are using routable IP addresses.
  • The PKS control plane is deployed inside of the NSX-T network. Both the PKS control plane components (VMs) and the Kubernetes Nodes use routable IP addresses.

I used William Lam’s excellent series on PKS with NSX-T to configure a lot of the settings, so I am going to assume a familiarity with that series. If not, I suggest you start there to get a good understanding of how everything is laid out.

Check PKS Prerequisites

Check the Ops Manager version - PKS 1.1 requires Ops Manager 2.1, you can view your current version by accessing the Ops Manager web interface and the version is listed at the bottom. If it’s not up to date enough, follow the Ops Manager upgrade documentation.

Ops Manager version

If you chose to upgrade deployed k8s nodes as part of the upgrade you will need to make sure you have at least 3 worker nodes and your workloads are spread across the nodes to ensure workloads are available. Check if PKS k8s deployments are correctly sized, and scale if it’s required (it’ll go red under the Status tab if it’s under pressure!)

Check PKS metrics PKS under pressure

Download the binaries

Download the necessary binaries from the Pivotal Container Service page - mine is vSphere. For the PKS and Kubectl API, download the binary for your “PKS Client” platform - I’m using Linux. Also download the updated Stemcell (linked on the right under Release Details), I am using the Ubuntu Trusty for vSphere one.

  • pivotal-container-service-1.1.0-build.311.pivotal
  • bosh-stemcell-3586.24-vsphere-esxi-ubuntu-trusty-go_agent.tgz
  • pks-linux-amd64-1.1.0-build.301
  • kubectl-linux-amd64-v1.10.3
Download PKS

Upgrade PKS Tile

From the Ops Manager home page, click on “Import a Product” and select the pivotal-container-service-1.1.0-build.311.pivotal - it’s a 3.7GB file, so it may take a while to update. Once it’s uploaded, click the + symbol to add the updated version to the Installation Dashboard.

PKS added

The Pivotal Container Service tile will turn orange to indicate it has changes, and a link to show that an updated stemcell is required appears. You can click on this link, or the STEMCELL LIBRARY link to upload the new stemcell.

PKS Upgraded

Click IMPORT STEMCELL and select bosh-stemcell-3586.24-vsphere-esxi-ubuntu-trusty-go_agent.tgz to import the new stemcell

Import Stemcell

Finally, apply the stemcell to the new Pivotal Container Service product.

Apply stemcell

Configure the PKS tile

The PKS tile will now be orange, but the changes can’t be applied until some extra configuration is added - click on the tile to open the Settings page

Under the Networking page, we have to add a certificate for the NSX user, move to an IP Block for the k8s node IP addressing, and add a DNS server configuration.

Generate a Super User Principal Identity

First up, we need create a certificate that will “represents a principal identity with superuser permissions that the PKS VM will use to communicate with NSX-T”. Follow the instructions in this document (6.1) https://docs.vmware.com/en/VMware-Pivotal-Container-Service/1.1/vmware-pks-11/GUID-PKS11-installing-nsx-t.html. I ran the following on my pks-client Ubuntu server.

Firstly, export some variables to use in the commands:

NODE_ID=$(cat /proc/sys/kernel/random/uuid)

Now generate a private key and certificate:

openssl req \
-newkey rsa:2048 \
 -x509 \
 -nodes \
 -new \
 -subj /CN=pks-nsx-t-superuser \
 -extensions client_server_ssl \
 -config <(
 cat /etc/ssl/openssl.cnf \
 <(printf '[client_server_ssl]\nextendedKeyUsage = clientAuth\n')
 ) \
 -sha256 \
 -days 730

Generating a 2048 bit RSA private key



writing new private key to ‘pks-nsx-t-superuser.key’


Create a variable with the certificate API request:

cert_request=$(cat <<END
"display_name": "$PI_NAME",
"pem_encoded": "$(awk '{printf "%s\\n", $0}' $NSX_SUPERUSER_CERT_FILE)"

Then POST the request to the NSX API:

curl -k -X POST \
"https://${NSX_MANAGER}/api/v1/trust-management/certificates?action=import" \
-H 'content-type: application/json' \
-d "$cert_request"

You can validate the certificate is installed in the NSX-T Manager interface under System > Trust > Certificates

Certificate Trust

Copy the “id” value from the response of the previous API call and create a variable:


Then create a variable to hold the API call to register the principal:

pi_request=$(cat <<END
"display_name": "$PI_NAME",
"name": "$PI_NAME",
"permission_group": "superusers",
"certificate_id": "$CERTIFICATE_ID",
"node_id": "$NODE_ID"

And POST the call to NSX-T Manager to create it:

curl -k -X POST \
"https://${NSX_MANAGER}/api/v1/trust-management/principal-identities" \
-H 'content-type: application/json' \
-d "$pi_request"

Finally, validate the certificate and key by using them to make a simple GET query on the API:

curl -k -X GET \
"https://${NSX_MANAGER}/api/v1/trust-management/principal-identities" \
--cert $(pwd)/"$NSX_SUPERUSER_CERT_FILE" \
--key $(pwd)/"$NSX_SUPERUSER_KEY_FILE"

Add the Certificate and Key to the PKS Tile

Back in Ops Manager, click on the Pivotal Container Service tile and select the Networking settings tab. Paste the contents of the .crt file into the Certificate field, and the .key contents into the Key field.

PKS Networking Settings

Configure the IP Block ID, Nodes DNS, and NAT mode

Next, further down the Networking page we need to add an NSX-T IP Block ID for the k8s nodes. If, like me, you followed William Lam’s configuration, part of the Ops Manager/BOSH setup was to configure a “Service Network” for the k8s nodes, on the subnet. In PKS 1.0 this Service Network was used to configure the IP addresses for the Kubernetes nodes. In PKS 1.1 the IP Block is used, with the first IP being assigned to the t1 router for the k8s cluster.

With this configuration (thanks to Francis Guillier for his help to understand):

  • k8s clusters deployed with PKS 1.0 will still live on service network (with a scale up event, for instance, new k8s worker nodes will be connected to this service network
  • new K8s clusters will be deployed using the Node IP Block (/24 allocated to this cluster)

In order to complete the upgrade, I needed to create a new IP Block in NSX-T that mirrors the Service Network:


Copy/paste the ID from NSX-T into the Nodes IP Block ID, disable the NAT mode checkbox and enter a DNS server IP for the Kubernetes Nodes.

PKS Network Config

Finally, click Save at the bottom of the page!

Configure the CEIP and Telemetry Options

CEIP Options

Validate the Errands Configuration

Last but not least, validate that the Post-Deploy Errands for NSX-T Validation, and Upgrade all clusters are set to On - this ensures NSX-T is configured correctly, and that all Kubernetes Nodes are upgraded.

PKS Validation Errands

Save the configuration and go back to the Installation Dashboard

Running the Upgrade

From the installation Dashboard, click “Apply changes” to begin the upgrade process!

Ops Manager Apply Changes

Depending on your environment this can take a while - you can follow along by showing the verbose output on the Applying Changes page

PKS Applying Changes

Upgrading the PKS and Kubectl CLIs

Upgrading the CLIs is simply a case of uploading the binaries, chmod to allow execution, then moving to the correct folder:

sudo chmod +x pks-linux-amd64-1.1.0-build.301
sudo mv pks-linux-amd64-1.1.0-build.301 /usr/local/bin/pks
sudo chmod +x kubectl-linux-amd64-v1.10.3
sudo mv kubectl-linux-amd64-v1.10.3 /usr/local/bin/kubectl

You can then validate the installed version with kubectl version

Share this post