VMware Event Broker Appliance - Certificates

Updating the TLS Certificate on VEBA

By default, the VMware Event Broker Appliance generates a self-signed TLS certificate that is used to support different web endpoints running on the appliance such as Stats (/stats), Status (/status), Logs (/bootstrap) and Events (/events). This will cause browsers to show the certificate as untrusted.

For organizations that require the use of a TLS certificate from a trusted authority, the VMware Event Broker Appliance provides an option for users to provide their certificate information during the OVF property configuration when deploying the virtual appliance.

In order to use a certificates from a trusted authority, please follow the steps outlined below.

Assumptions

Certificates from a trusted authority pre-downloaded onto your local desktop
- The public/private key pair must exist before hand. The public key certificate must be .PEM encoded and match the given private key.

Steps

In the example, the private key file is named privateKey.key and certificate file is named certificate.crt

Encode both the private key and the certificate file using base64 encoding.

Microsoft Windows (PowerShell)

$privateKeyContent = Get-Content -Raw privateKey.key
$privateKeybase64 = [System.Convert]::ToBase64String([System.Text.Encoding]::ASCII.GetBytes($privateKeyContent))
Write-Host "Encoded Private Key:`n$privateKeybase64`n"

$certContent = Get-Content -Raw certificate.crt
$certbase64 = [System.Convert]::ToBase64String([System.Text.Encoding]::ASCII.GetBytes($certContent))
Write-Host "Encoded Certificate:`n$certbase64`n"

Encoded Private Key:
LS0tLS1CRUd......==


Encoded Certificate Key:
LS0tLS1CRUe......==

MacOS/Linux

cat privateKey.key | base64

LS0tLS1CRUd......==

cat certificate.crt | base64

LS0tLS1CRUd......==

Using the output from the previous step, the base64 content can now be provided in Custom TLS Certificate Configuration section of the OVF property during the deployment of the VMware Event Broker Appliance.
- Custom VMware Event Broker Appliance TLS Certificate Private Key (Base64)
- Custom VMware Event Broker Appliance TLS Certificate Authority Certificate (Base64)
Power on the VMware Event Broker Appliance and ensure that the provided TLS certificate is now used instead of the auto-generated self-sign TLS certificate by opening a browser to one of the VMware Event Broker Appliance endpoints such as /status.

Replacing an Existing Cert on VEBA

If you need to replace the default self-signed certificate, or replace an expired certificate, follow the steps outlined below.

Assumptions

Your SSL certificate must match the FQDN that you picked when you deployed VEBA
Pre-download your public and private key from your Certificate Authority, which must be .PEM encoded.

Steps

Step 1 - Transfer or copy the contents of the root certificate to the VMware Event Broker Appliance. If you need a free, publicly signed certificate, see the Let’s Encrypt section below.

Step 2 - Delete the existing Event Router TLS secret

kubectl -n vmware-system delete secret eventrouter-tls

Step 3 - Create a new Event Router TLS secret with the new certificate keypair.

kubectl -n vmware-system create secret tls eventrouter-tls --key privkey.pem --cert pubkey.pem

Step 4 - Delete the existing Contour TLS secret

kubectl -n contour-external delete secret default-cert

Step 5 - Create a new Contour TLS secret with the new certificate keypair.

kubectl -n contour-external create secret tls default-cert --key privkey.pem --cert pubkey.pem

Step 6 - Check for valid HTTPproxy status

kubectl -n vmware-system get httpproxy

Expected output:

NAME           FQDN               TLS SECRET        STATUS   STATUS DESCRIPTION
event-router   veba.domain.com    eventrouter-tls   valid    Valid HTTPProxy

Adding a Trusted Root Certificate to VEBA

For organizations that require the use of a Private or Enterprise Certificate Authority (CA) and wish to have full TLS trust when the VMware Event Broker Appliance is configured with a vCenter Server that has a TLS certificate generated by the internal CA, a root of trust must be established.

To support this use case, a post-deployment reconfiguration of the VMware Event Broker Appliance is required to import the trusted root certificate of your internal certificate authority. If this is not performed, you may see the following warning in the VMware Event Router logs when connecting to the configured vCenter Server.

WARN    [VCENTER]       vcenter/vcenter.go:112  using potentially insecure connection to vCenter  {"address": "https://vcsa.primp-industries.local", "insecure": true}

In order to import your trusted root certificate, please follow the steps outlined below.

Assumptions

Pre-download your root certificates from your internal certificate, which must be .PEM encoded

In the example, the downloaded root certificate file is named ca-root.crt

Steps

Step 1 - Transfer or copy the contents of the root certificate to the VMware Event Broker Appliance and create a new kubernetes configMap from that file.

kubectl -n vmware-system create cm ca-root-cert --from-file ca-root.crt

Step 2 - Undeploy the VMware Event Router and delete the current configuration secret.

kubectl -n vmware-system delete deploy vmware-event-router
kubectl -n vmware-system delete secret event-router-config

Step 3 - Edit /root/config/event-router-k8s.yaml and append the additional volumenMount and volume entry as shown in the snippet below.

snip
...
          volumeMounts:
            - name: config
              mountPath: /etc/vmware-event-router/
              readOnly: true
            - name: ca-root-cert
              mountPath: /etc/vmware-event-router/ssl
              readOnly: true
      volumes:
        - name: config
          secret:
            secretName: event-router-config
        - name: ca-root-cert
          configMap:
            name: ca-root-cert

Step 4 - In the same file (step 3) append the additional certificates section to the end of the file as shown in the snippet below.

snip
...
metricsProvider:
  default:
    bindAddress: 0.0.0.0:8082
  name: veba-metrics
  type: default
certificates:
  rootCAs:
    - /etc/ssl/certs/ca-certificates.crt
    - /etc/vmware-event-router/ssl/ca-root.crt

Note: If insecureSSL is set to true, please update that to false which will now ensure TLS verification is performed when connecting to your vCenter Server.

Step 5 - Recreate the VMware Event Router configuration and redeploy the VMware Event Router deployment.

kubectl -n vmware-system create secret generic event-router-config --from-file=/root/config/event-router/vmware-event-router-config-vcenter.yaml
kubectl -n vmware-system apply -f /root/config/event-router-k8s.yaml

Step 6 - Verify the VMware Event Router pod is running successfully.

kubectl -n vmware-system get pods
NAME                                   READY   STATUS    RESTARTS   AGE
tinywww-dd88dc7db-q6pzw                1/1     Running   0          166m
veba-rabbit-server-0                   1/1     Running   0          167m
veba-ui-8d6584f84-ckvwb                1/1     Running   0          166m
vmware-event-router-7759d8bffc-45c92   1/1     Running   0          37m

Lastly, you can also look at the VMware Event Router log and ensure that you no longer see the message potentially insecure connection to vCenter, which is what you would see if the connection was not trusted if you had TLS verified disabled. The following DEBG log entry should appear when the VMware Event Router is deployed with the debug flag.

kubectl -n vmware-system logs vmware-event-router-7759d8bffc-kt2jm

DEBUG   [VCENTER]       vcenter/vcenter.go:136  setting custom root CAs {"certificates": "/etc/ssl/certs/ca-certificates.crt:/etc/vmware-event-router/ssl/ca-root.crt"}

Obtaining a Signed SSL certificate from Let’s Encrypt

This section demonstrates installation of the Let’s Encrypt Certbot Docker image onto the VEBA appliance, then uses DNS validation to verify domain ownership.

Steps

Step 1 - Start the Docker daemon. VEBA uses containerd for its container runtime - the Docker daemon is disabled by default.

systemctl start docker

Step 2 - Pull the Certbot Docker image

docker pull certbot/certbot

Step 3 - Run certbot. For the -d (domain) switch, use your VEBA FQDN. You will be prompted for an e-mail address as well as some yes/no questions.

 docker run -it --rm --name certbot -v "/etc/letsencrypt:/etc/letsencrypt" `
 -v "/var/lib/letsencrypt:/var/lib/letsencrypt" `
 -v "/var/log/letsencrypt:/var/log/letsencrypt" `
 certbot/certbot certonly --manual -d veba02.vmweventbroker.io --preferred-challenges dns

Enter email address (used for urgent renewal and security notices)
 (Enter 'c' to cancel): [email protected]

 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Please read the Terms of Service at
https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf. You must
agree in order to register with the ACME server. Do you agree?
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
(Y)es/(N)o: Y

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Would you be willing, once your first certificate is successfully issued, to
share your email address with the Electronic Frontier Foundation, a founding
partner of the Let's Encrypt project and the non-profit organization that
develops Certbot? We'd like to send you email about our work encrypting the web,
EFF news, campaigns, and ways to support digital freedom.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
(Y)es/(N)o: n
Account registered.
Requesting a certificate for veba02.vmweventbroker.io

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Please deploy a DNS TXT record under the name:

_acme-challenge.veba02.vmweventbroker.io.

with the following value:

KfqRahey6wChxY_cZgNbRAlRpS34KQhjvQvTXnzRXgo

Before continuing, verify the TXT record has been deployed. Depending on the DNS
provider, this may take some time, from a few seconds to multiple minutes. You can
check if it has finished deploying with aid of online tools, such as the Google
Admin Toolbox: https://toolbox.googleapps.com/apps/dig/#TXT/_acme-challenge.veba02.vmweventbroker.io.
Look for one or more bolded line(s) below the line ';ANSWER'. It should show the
value(s) you've just added.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Press Enter to Continue

Step 4 - Using your public DNS provider’s tools, configure the required TXT record as prompted in Step 2.

Step 5 - Press Enter to continue. If you have configured DNS properly, the certificate PEM files will be saved in the location specified.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Press Enter to Continue

Successfully received certificate.
Certificate is saved at: /etc/letsencrypt/live/veba02.vmweventbroker.io/fullchain.pem
Key is saved at:         /etc/letsencrypt/live/veba02.vmweventbroker.io/privkey.pem
This certificate expires on 2022-04-12.
These files will be updated when the certificate renews.

NEXT STEPS:
- This certificate will not be renewed automatically. Autorenewal of --manual certificates requires the use of an authentication hook script (--manual-auth-hook) but one was not provided. To renew this certificate, repeat this same certbot command before the certificate's expiry date.

Step 6 - Install the certificate - follow the instructions starting with step 2 of Replacing an Existing Cert on VEBA. Note from the output above that the public key file is named fullchain.pem - you will need to pass this value for the --cert argument when creating the Kubernetes TLS certificates.

Step 7 - Stop the Docker daemon

systemctl stop docker

Step 8 (optional) - If you want to automate renewals, this is an excellent blog on configuring automated certificate renewals using DNS validation.

Replacing the default self-signed TLS certificate in VMware Event Broker Appliance.

Documentation

Updating the TLS Certificate on VEBA

Assumptions

Steps

Replacing an Existing Cert on VEBA

Assumptions

Steps

Adding a Trusted Root Certificate to VEBA

Assumptions

Steps

Obtaining a Signed SSL certificate from Let’s Encrypt

Steps