Rotating Operations Manager Root CA with Custom CA certificate

Recently a customer reported an issue with harbor UAA login after using custom certificates and we identified the root cause was the mismatch in the root ca of harbor and opsmanager. By default, the harbor vm root CA is same as the Opsman root CA. We suggested the customer to get the opsman root ca updated which resolved the issue. In this blog post, I will take you through how we can replace the opsman root ca as I felt it is a little confusing.

Warning: You must complete these steps in the exact order specified. Otherwise, you may damage your deployment

Part 1 : Retrieve UAA token

To access the Ops Manager API, you must authenticate to the Ops Manager User Account and Authentication (UAA) server.

To retrieve the authorization token from internal UAA on vSphere, do the following:

Step 1. SSH onto the Ops Manager VM 
 ssh ubuntu@OPS-MANAGER-FQDN
Step 2: Retrieve UAA token

Retrieve your UAA token by running the below command.

uaac token owner get

For example: Internal UAA

ubuntu@opsman-corp-local:~$ uaac token owner get
 Client ID: opsman(default)
 Client secret:(leave blank)
 User name: admin
 Password: (admin user password of opsman)

 Successfully fetched token via owner password grant.
 Target: https://opsman.corp.local/uaa
 Context: admin, from client opsman

Note: Depending on whether your Ops Manager UAA is internal or external, run a command to retrieve your UAA token and respond to the authentication prompts. For more details, refer Pivotal doc.

Internal: You must run Ops Manager API commands from the Opsman VM.
External: You may run commands from your local machine.

Step 3: List Tokens

List your tokens by running the following command.

uaac contexts
For Example:
ubuntu@opsman-corp-local:~$ uaac context
 [0]*[https://opsman.corp.local/uaa]
   skip_ssl_validation: true
 [0]*[admin]
       user_id: b0d21880-7fc5-41b6-84c1-7406345b9fd7
       client_id: opsman
       access_token: eyJhbGciOiJSUzI1NiIsImtpZCI6ImtleS0xIiwidHlwIjoiSldUIn0.eyJqdGkiOFD-Mp56Cl3KiXhDodhKPXbNx_rs4KFYJGBFvSDzQ
       token_type: bearer
       refresh_token: eyJhbGciOiJSUzI1NiIsImtpZCI6ImtleS0xIiwidHlwIjoiSldUIn0.eyJqdGkiOc1t2OvclzQ7CPemIDSpUK7lIpSNKTBDPxEGwLeZCDOt31v-
       expires_in: 43199
       scope: opsman.admin scim.me uaa.admin clients.admin
       jti: 86f94a58d8144d1b802d4844690a224c

Make a note of access_token in the above example given. That is your UAA token.

Note: Token and certificates in the examples given in the blog has been edited for the better view

Step 4: List the current CA

Run the below API to list the current root CA

curl "https://OPS-MAN-FQDN/api/v0/certificate_authorities" \ 
-X GET \ 
-H "Authorization: Bearer YOUR-UAA-ACCESS-TOKEN" -k |jq
For Example:
ubuntu@opsman-corp-local:~$ curl "https://opsman-corp-local/api/v0/certificate_authorities" -X GET -H "Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6ImtleS0xIiwidHlwIjoiSldUIn0.eyJqdGkiOiI4NmY5NGE1OGQ4MTQ0ZDFiODAyZDQ4NDQ2OTBhM" -k |jq
   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                  Dload  Upload   Total   Spent    Left  Speed
 100  2995    0  2995    0     0   5577      0 --:--:-- --:--:-- --:--:--  5587
 {
   "certificate_authorities": [
     {
       "guid": "5fe5cda1f54485192420",
       "issuer": "Pivotal",
       "created_on": "2019-01-24T20:58:24Z",
       "expires_on": "2023-01-24T20:58:24Z",
       "active": true,
       "cert_pem": "-----BEGIN CERTIFICATE-----\nMIIDUDCCAjigAwIBAgIUD2xtpVHKN22v3hjF0xNKJvqT3/YwDQYJKoZIhvcNAQE637XWBO7CU\nt23Vd0W6CevXwWh6/2GPMKHmyoZk5B2ENaIQh1vKr4PDBtzC\n-----END CERTIFICATE-----\n"

Make a note of the issuer and active status in the above example. You can see that the root ca is the default one, which is issued by Pivotal.

Part 2: Rotate the Opsman root CA

To rotate the Ops Manager root CA and leaf certificates, perform the steps. For any additional details, please refer to Pivotal document.

Step 1: Add a new Root CA

To use your own custom CA, run the following API

curl "https://OPS-MAN-FQDN/api/v0/certificate_authorities" \ 
-X POST \ 
-H "Authorization: Bearer YOUR-UAA-ACCESS-TOKEN" \ 
-H "Content-Type: application/json" \ 
-d '{"cert_pem": "-----BEGIN CERTIFICATE-----\YOUR-CERTIFICATE", "private_key_pem": "-----BEGIN RSA PRIVATE KEY-----\YOUR-KEY"}'

Replace the below in the command with actual data

  • OPS-MAN-FQDN
  • YOUR-UAA-ACCESS-TOKEN (retrieved in Part 1)
  • YOUR-CERTIFICATE (Root CA certificate)
  • YOUR-KEY (Root CA key)
For Example: 
curl "https://opsman-corp-local/api/v0/certificate_authorities" \
 -X POST \
 -H "Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6ImtleS0xIiwidHlwIjoiSldUIn0.eyJqdGkiOFD-Mp56Cl3KiXhDodhKPXbNx_rs4KFYJGBFvSDzQ" \
 -H "Content-Type: application/json" \
 -d '{"cert_pem": "-----BEGIN CERTIFICATE-----\nMIID5TCCAs2gAwIBAgIJALBqH1TyW5p0MA0GCSqGSIb3DPdVJbbdcaYFDRAqIQj8g95GQjOUVqkqT\nZrtI8sHpvqn6i+8RYQ2i4i87UyrjF/hzWO6terkmOHRMpI/cClTNu0s=\n-----END CERTIFICATE-----", "private_key_pem": "-----BEGIN RSA PRIVATE KEY-----\nMIIEogIBAAKCAQEAyGUtkkE39emy8y5j1pk21B0TMhULCiLQ6gAtwf5WxyI2odLq\nxt3/nBC+UpZOpOERlE\nt3dcpx9/HxrH3Egcf7vZ/3f/eQSuIS01oL7PWwd87OjU19bhPqA=\n-----END RSA PRIVATE KEY-----"}' -k

Note: The certificate has to be converted to single line. You can use some tool to do the same or simply append \n to each line like I did in the example to put them together in a line.

Step 2: List the root CA

Now, lets confirm that the root ca we just added is available. You can run the same API mentioned in Part 1, step 4 to list the root CA.

For Example:
ubuntu@opsman-corp-local:~$ curl "https://opsman-corp-local/api/v0/certificate_authorities" \
   -X GET \
   -H "Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6ImtleS0xIiwidHlwIjoiSldUIn0.eyJqdGkiOFD-Mp56Cl3KiXhDodhKPXbNx_rs4KFYJGBFvSDzQ" -k | jq
   % Total % Received % Xferd Average Speed Time Time Time Current
   Dload Upload Total Spent Left Speed
   100 2995 0 2995 0 0 15164 0 --:--:-- --:--:-- --:--:-- 15280
   {
   "certificate_authorities": [
   {
   "guid": "5fe5cda1f54485192420",
   "issuer": "Pivotal",
   "created_on": "2019-01-24T20:58:24Z",
   "expires_on": "2023-01-24T20:58:24Z",
   "active": true,
   "cert_pem": "-----BEGIN CERTIFICATE-----\nMIIDUDCCAjigAwIBAgIUD2xtpVHKN22v3hjF0xNKJvqT3/vlsgYUpiLxbtnGdpzGmGVw\noJsU228dmcQENaIQh1vKr4PDBtzC\n-----END CERTIFICATE-----\n"
   },
   {
   "guid": "d81098ee2ab56ad1a13d", 
   "issuer": "corp.local",
   "created_on": "2019-10-15T08:43:50Z",
   "expires_on": "2029-10-12T05:48:57Z", 
   "active": false,
   "cert_pem": "-----BEGIN CERTIFICATE-----\nMIID5TCCAs2gAwIBAgIJALBqH1TyW5p0MA0GCSqGSIb3DQEBCwUAMIGIMQswCQYD\O6terkmOHRMpI/cClTNu0s=\n-----END CERTIFICATE-----"

Identify your newly added CA, which has active set to false. Record its guid.

Now run apply changes from opsman by following the below steps before going to Step 3.

  1. Navigate to https://OPS-MAN-FQDN in a browser and log in to Ops Manager
  2. Click the BOSH Director tile in the Installation Dashboard
  3. Select the Director Config pane
  4. Select Recreate All VMs. This propagates the new CA to all VMs to prevent downtime
  5. Go back to the Installation Dashboard. For each service tile you have installed, do the following: Click the tile.
    • Click the Errands tab.
    • Uncheck Upgrade all clusters errand
    • Navigate to Ops Manager, click Review Pending Changes
    • Click Apply Changes
Step 3: Activate the new root ca

To activate the newly added custom root ca, run the below API

curl "https://OPS-MAN-FQDN/api/v0/certificate_authorities/CERT-GUID/activate" \ 
-X POST \ 
-H "Authorization: Bearer YOUR-UAA-ACCESS-TOKEN" \ 
-H "Content-Type: application/json" \ 
-d '{}'

Note: Update the Opsman FQDN and the CERT-GUID in the above API

For Example:
curl "https://opsman-corp-local/api/v0/certificate_authorities/d81098ee2ab56ad1a13d/activate" \
 -X POST \
 -H "Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6ImtleS0xIiwidHlwIjoiSldUIn0.eyJqdGkiOFD-Mp56Cl3KiXhDodhKPXbNx_rs4KFYJGBFvSDzQ" \
 -H "Content-Type: application/json" \
 -d '{}' -k
Step 4: List the root CA

Once again list the root CA and confirm the newly added custom root CA is active.

For Example:
curl "https://opsman-corp-local/api/v0/certificate_authorities" -X GET -H "Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6ImtleS0xIiwidHlwIjoiSldUIn0.eyJqdGkiOFD-Mp56Cl3KiXhDodhKPXbNx_rs4KFYJGBFvSDzQ" -k | jq
 100 2995 0 2995 0 0 1375 0 --:--:-- 0:00:02 --:--:-- 1377
 {
 "certificate_authorities": [
 {
 "guid": "5fe5cda1f54485192420",
 "issuer": "Pivotal",
 "created_on": "2019-01-24T20:58:24Z",
 "expires_on": "2023-01-24T20:58:24Z",
 "active": false,
 "cert_pem": "-----BEGIN CERTIFICATE-----\nMIIDUDCCAjigAwIBAgIUD2xtpVHKN22v3hjF0xNKJvqT3/YwDQYJLxbtnGdpzGmO7CU\nt23Vd0W6CevXwWh6/2GPMKHmyoZk5B2ENaIQh1vKr4PDBtzC\n-----END CERTIFICATE-----\n"
 },
 {
 "guid": "d81098ee2ab56ad1a13d",
 "issuer": "corp.local",
 "created_on": "2019-10-15T08:43:50Z",
 "expires_on": "2029-10-12T05:48:57Z", 
 "active": true,
 "cert_pem": "-----BEGIN CERTIFICATE-----\nMIID5TCCAs2gAwIBAgIJALBqH1TyW5p0MA0GCSqGSeEX5yPdVJbbdcaYFDRAqIQS8g95GQjOUVqkqT\nZrtI8sHpvqn6i+8RYQ2i4i87UyrjF/hzWO6terkmOHRMpI/cClTNu0s=\n-----END CERTIFICATE-----"

In the above example, you can see that the corp.local is the active CA now.

Step 4: Rotate Non-Configurable Leaf Certificates

After activating the new root CA, you must rotate non-configurable leaf certificates from the root CA. Use curl to make an API call to regenerate all non-configurable certificates and apply the new CA to your existing BOSH Director

curl "https://OPS-MAN-FQDN/api/v0/certificate_authorities/active/regenerate" \     
-X POST \     
-H "Authorization: Bearer YOUR-UAA-ACCESS-TOKEN" \     
-H "Content-Type: application/json" \     
-d '{}'

To complete the process, run the below steps as well.

  1. Navigate to https://OPS-MAN-FQDN in a browser and log in to Ops Manager
  2. Click the BOSH Director tile in the Installation Dashboard.
  3. Select the Director Config pane.
  4. Select Recreate All VMs. This propagates the new CA to all VMs to prevent downtime.
  5. Go back to the Installation Dashboard. For each service tile you have installed, do the following:
    • Click the tile.
    • Click the Errands tab.
    • Uncheck Upgrade all clusters errand
    • Navigate to Ops Manager, click Review Pending Changes
    • Click Apply Changes to perform a second redeploy.
Step 5: Delete the old CA

To delete the old default ca, you can run the below API. Please note, this step is optional.

To delete the CA, run the below API to list your root CAs and retrieve the GUID of the old default CA. In this example given, the GUID is 5fe5cda1f54485192420

Run the below API to delete the old root ca

curl "https://OPS-MAN-FQDN/api/v0/certificate_authorities/OLD-CERT-GUID" \ 
-X DELETE \ 
-H "Authorization: Bearer YOUR-UAA-ACCESS-TOKEN"

Note: OLD-CERT-GUID is the GUID of your old, inactive CA

Step 6: Recreate Bosh deployment

Recreate the bosh deployment for PKS clusters by running the below command. 

 bosh -d service_instance_xxxx_xxxx_xxxx_xxxxxxxx recreate

Note: Based on how your pods are designed, you may face downtime on the services running while recreating the deployment. Hence I suggest to schedule maintenance window for this activity.

Step 7: Apply changes

Go to OpsMan installation dashboard and apply changes one last time to confirm all is well.
Make sure this time to perform the below, before applying changes

  • Uncheck : Bosh tile > Director config tab > Recreate All VMs
  • Enable : “Upgrade all clusters errand” on PKS tile

Note: I suggest you try this out in your test environment before running it in production. To be on the safer side, reach out to VMware PKS support team for help

Leave a Reply

Fill in your details below or click an icon to log in:

Gravatar
WordPress.com Logo

You are commenting using your WordPress.com account. ( Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. ( Log Out /  Change )

Cancel

Connecting to %s

Create a website or blog at WordPress.com

Up ↑

%d bloggers like this: