Inject license via Hashicorp Vault Agent Annotations | HiveMQ Platform Operator (new)

Vault Agent Sidecar Injector service leverages the sidecar container pattern and Kubernetes mutating admission webhook to intercept pods that define specific annotations and inject a Vault Agent container to manage these secrets.

This is beneficial because:

Applications remain Vault unaware as the secrets are stored on the file-system in their container.
Existing deployments require no change; as annotations can be patched.
Access to secrets can be enforced via Kubernetes service accounts and namespaces

In this tutorial, you setup Vault and this injector service with the Vault Helm chart. Then you will deploy several applications to demonstrate how this new injector service retrieves and writes these secrets for the applications to use.

Prerequisites

These instructions require the following tools on the local machine:

Kubernetes command-line interface (CLI)
Helm CLI

The kubectl context should be set to the Kubernetes cluster where the HiveMQ broker will be installed.

Instructions

Install the Hashicorp Vault Helm Chart

Add the HashiCorp Helm repository.
helm repo add hashicorp https://helm.releases.hashicorp.com
Update all the repositories to ensure helm is aware of the latest versions.
helm repo update hashicorp
Install the latest version of the Hashicorp Vault server running in development mode.
Development mode: Running a Hashicorp Vault server in development is automatically initialized and unsealed. This is ideal in a learning environment but NOT recommended for a production environment.
helm install vault hashicorp/vault --set "server.dev.enabled=true"
The vault pod and vault Agent Injector pod are deployed in the default namespace.
Display all the pods in the default namespace.
kubectl get pods
NAME READY STATUS RESTARTS AGE vault-0 1/1 Running 0 80s vault-agent-injector-5945fb98b5-tpglz 1/1 Running 0 80s
The vault-0 pod runs a vault server in development mode. The vault-agent-injector pod performs the injection based on the annotations present or patched on a deployment.
Wait until the vault-0 pod and vault-agent-injector pod are running and ready (1/1).

Set a secret in Hashicorp Vault

Copy the hivemq-license file to the vault-0 pod.
kubectl cp hivemq4.lic vault-0:/tmp/
Verify that the file is copied.
kubectl exec -it vault-0 -- ls /tmp
hivemq4.lic
Start an interactive shell session on the vault-0 pod.
kubectl exec -it vault-0 -- /bin/sh

Your system prompt is replaced with a new prompt / $. Commands issued at this prompt are executed on the vault-0 container.
Enable kv-v2 secrets at the path hivemq.
vault secrets enable -path=hivemq kv-v2
Success! Enabled the kv-v2 secrets engine at: hivemq/
Create a secret at path hivemq/test/license with a hivemq_license_b64 key and base64-encoded /tmp/hivemq4.lic file.
cd /tmp vault kv put hivemq/test/license hivemq_license_b64="$(base64 -w 0 hivemq4.lic)"
====== Secret Path ====== hivemq/data/test/license ======= Metadata ======= Key Value --- ----- created_time 2024-02-21T17:34:39.261249639Z custom_metadata <nil> deletion_time n/a destroyed false version 1
Verify that the secret is defined at the path hivemq/test/license.
vault kv get hivemq/test/license
====== Secret Path ====== hivemq/data/test/license ======= Metadata ======= Key Value --- ----- created_time 2024-02-21T14:57:01.446984026Z custom_metadata <nil> deletion_time n/a destroyed false version 1 ========= Data ========= Key Value --- ----- hivemq_license_b64 SCFNUSRbM10.......
The secret is ready for the application.
Lastly, exit the vault-0 pod.
exit

Configure Kubernetes authentication

Hashicorp Vault provides a Kubernetes authentication method that enables clients to authenticate with a Kubernetes Service Account Token. This token is provided to each pod when it is created.

Start an interactive shell session on the vault-0 pod.
kubectl exec -it vault-0 -- /bin/sh

Your system prompt is replaced with a new prompt / $. Commands issued at this prompt are executed on the vault-0 container.
Enable the Kubernetes authentication method.
vault auth enable kubernetes
Success! Enabled kubernetes auth method at: kubernetes/
Hashicorp Vault accepts a service token from any client in the Kubernetes cluster. During authentication, Hashicorp Vault verifies that the service account token is valid by querying a token review Kubernetes endpoint.
Configure the Kubernetes authentication method to use the location of the Kubernetes API.
Note: For the best compatibility with recent Kubernetes versions, ensure you are using Hashicorp Vault v1.13.3 or greater.
vault write auth/kubernetes/config kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443"
Successful output from the command resembles this example:
Success! Data written to: auth/kubernetes/config
The environment variable KUBERNETES_PORT_443_TCP_ADDR is defined and references the internal network address of the Kubernetes host.
For a client to read the secret data defined at hivemq/test/license, requires that the read capability be granted for the path hivemq/data/test/license. A policy defines a set of capabilities.
Write out the policy named hivemq that enables the read capability for secrets at path hivemq/data/test/license.
vault policy write hivemq - <<EOF path "hivemq/data/test/license" { capabilities = ["read"] } EOF
Create a Kubernetes authentication role named hivemq.
vault write auth/kubernetes/role/hivemq \ bound_service_account_names=hivemq-platform-pod-broker \ bound_service_account_namespaces=hivemq \ policies=hivemq \ ttl=24h
Successful output from the command resembles this example:
Success! Data written to: auth/kubernetes/role/hivemq
The role connects the Kubernetes service account, hivemq-platform-pod-broker, and namespace, hivemq, with the Hashicorp Vault policy, hivemq. The tokens returned after authentication are valid for 24 hours.
Lastly, exit the vault-0 pod.
exit

Inject secrets into the pods

If you do not have values.yaml file yet, you can get the latest version from the Helm chart repository and store it as a file:
helm show values hivemq/hivemq-platform > values-hivemq-platform-with-annotations.yaml
Edit the values-hivemq-platform-with-annotations.yaml file. Add annotations to the HiveMQ Pods.
nodes: # Annotations to add to the HiveMQ Pods annotations: vault.hashicorp.com/agent-inject: "true" vault.hashicorp.com/role: "hivemq-platform" vault.hashicorp.com/agent-inject-status: 'update' vault.hashicorp.com/agent-inject-secret-hivemq4.lic: "hivemq/data/test/license" vault.hashicorp.com/secret-volume-path-hivemq4.lic: "/opt/hivemq/license/" vault.hashicorp.com/agent-inject-template-hivemq4.lic: | {{- with secret "hivemq/data/test/license" -}} {{- $hivemq_broker_license := base64Decode .Data.data.hivemq_license_b64 -}} {{- $hivemq_broker_license -}} {{- end -}}
(Re)install hivemq
helm upgrade op --install hivemq/hivemq-platform-operator sleep 30 helm upgrade broker --install hivemq/hivemq-platform -n hivemq -f values-hivemq-platform-with-annotations.yaml --set nodes.replicaCount=1
Get all the pods in the hivemq namespace.
kubectl get pods --namespace hivemq
NAME READY STATUS RESTARTS AGE broker-0 2/2 Running 0 18m hivemq-op-bc-vbd25 1/1 Running 0 46m
Wait until the re-deployed hivemq pod reports that it is Running and ready (2/2).
This new pod now launches two containers. The application container, named hivemq, and the Hashicorp Vault Agent container, named vault-agent.
Display the logs of the vault-agent container in the new hivemq pod.
kubectl logs \ $(kubectl get pod -l hivemq-platform=broker -o jsonpath="{.items[0].metadata.name}") \ --container vault-agent
Hashicorp Vault Agent manages the token lifecycle and the secret retrieval. The secret is rendered in the hivemq container at the path /opt/hivemq/license/.
Display the secret written to the hivemq container.
kubectl exec \ $(kubectl get pod -l hivemq-platform=broker -o jsonpath="{.items[0].metadata.name}") \ --container hivemq -- cat /opt/hivemq/license/hivemq4.lic
The base64-decoded secret data is present on the container