Inject license via Hashicorp Vault Agent Init Container | HiveMQ Platform Operator (new)

Vault Agent provides a number of different helper features, specifically addressing the following challenges:

Automatic authentication
Secure delivery/storage of tokens
Lifecycle management of these tokens (renewal & re-authentication)

Vault Agent can help authenticate to Hashicorp Vault, retrieving a base64 encoded HiveMQ license, decoding the license, and saving it to a file in the HiveMQ container.

1 Prerequisites
2 Instructions
3 Related articles

Prerequisites

These instructions require the following tools on the local machine:

Kubernetes command-line interface (CLI)
Helm CLI

The kubectl context should be set to the Kubernetes cluster where the HiveMQ broker will be installed.

Instructions

Install the Hashicorp Vault Helm Chart

Add the HashiCorp Helm repository.
helm repo add hashicorp https://helm.releases.hashicorp.com
Update all the repositories to ensure helm is aware of the latest versions.
helm repo update hashicorp
Install the latest version of the Hashicorp Vault server running in development mode.
Development mode: Running a Hashicorp Vault server in development is automatically initialized and unsealed. This is ideal in a learning environment but NOT recommended for a production environment.
kubectl create namespace vault kubectl config set-context --current --namespace=vault helm install vault hashicorp/vault --set "server.dev.enabled=true" --namespace vault
The vault pod and vault Agent Injector pod are deployed in the default namespace.
Display all the pods in the default namespace.
kubectl get pods
Successful output from the command resembles this example:
NAME READY STATUS RESTARTS AGE vault-0 1/1 Running 0 80s vault-agent-injector-5945fb98b5-tpglz 1/1 Running 0 80s
The vault-0 pod runs a vault server in development mode. The vault-agent-injector pod performs the injection based on the annotations present or patched on a deployment.
Wait until the vault-0 pod and vault-agent-injector pod are running and ready 1/1.

Set a secret in Hashicorp Vault

Prepare your HiveMQ license in the file named hivemq.lic.
Run the command on the vault-0 pod to enable kv-v2 secrets at the path hivemq. The command is executed on the vault-0 container.
kubectl exec vault-0 -n vault -- vault secrets enable -path=hivemq kv-v2
Successful output from the command resembles this example:
Success! Enabled the kv-v2 secrets engine at: hivemq/
Create a new secret at path hivemq/test/license with a hivemq_license_b64 key and base64-encoded hivemq.lic file.
kubectl exec vault-0 -n vault -- vault kv put hivemq/test/license hivemq_license_b64="$(base64 < hivemq.lic)"
Successful output from the command resembles this example:
====== Secret Path ====== hivemq/data/test/license ======= Metadata ======= Key Value --- ----- created_time 2024-02-21T17:34:39.261249639Z custom_metadata <nil> deletion_time n/a destroyed false version 1
Verify that the secret is defined at the path hivemq/test/license.
kubectl exec vault-0 -n vault -- vault kv get hivemq/test/license
Successful output from the command resembles this example:
====== Secret Path ====== hivemq/data/test/license ======= Metadata ======= Key Value --- ----- created_time 2024-02-21T14:57:01.446984026Z custom_metadata <nil> deletion_time n/a destroyed false version 1 ========= Data ========= Key Value --- ----- hivemq_license_b64 SCFNUSRbM10.......
The secret is ready for the application.

Configure Kubernetes authentication

Hashicorp Vault provides a Kubernetes authentication method that enables clients to authenticate with a Kubernetes Service Account Token. This token is provided to each pod when it is created.

Enable the Kubernetes authentication method.
kubectl exec vault-0 -n vault -- vault auth enable kubernetes
Successful output from the command resembles this example:
Success! Enabled kubernetes auth method at: kubernetes/
Hashicorp Vault accepts a service token from any client in the Kubernetes cluster. During authentication, Hashicorp Vault verifies that the service account token is valid by querying a token review Kubernetes endpoint.
Configure the Kubernetes authentication method to use the location of your HiveMQ Kubernetes cluster.
Note: Ensure you are using Hashicorp Vault v1.13.3 or greater for the best compatibility with recent Kubernetes versions.
kubectl exec vault-0 -- vault write auth/kubernetes/config kubernetes_host=https://$(kubectl exec vault-0 -- env | grep KUBERNETES_PORT_443_TCP_ADDR | cut -d= -f2):443
Successful output from the command resembles this example:
Success! Data written to: auth/kubernetes/config
The environment variable KUBERNETES_PORT_443_TCP_ADDR is defined and references the internal network address of the Kubernetes host.
Verify the kubernetes_host of the created kubernetes authentication method.
kubectl exec vault-0 -- vault read auth/kubernetes/config
Key Value --- ----- disable_iss_validation true disable_local_ca_jwt false issuer n/a kubernetes_ca_cert n/a kubernetes_host https://10.0.0.1:443 pem_keys [] token_reviewer_jwt_set false use_annotations_as_alias_metadata false
For a client to read the secret data defined at hivemq/test/license, the read capability must be granted for the path hivemq/data/test/license. A policy defines a set of capabilities.
Create a new policy named hivemq that grants read access to secrets at the path hivemq/data/test/license.
vault policy write hivemq - <<EOF path "hivemq/data/test/license" { capabilities = ["read"] } EOF
Create a Kubernetes authentication role named hivemq.
vault write auth/kubernetes/role/hivemq \ bound_service_account_names=hivemq-platform-pod-broker \ bound_service_account_namespaces=hivemq \ policies=hivemq \ ttl=24h
Successful output from the command resembles this example:
Success! Data written to: auth/kubernetes/role/hivemq
The role connects the Kubernetes service account, hivemq-platform-pod-broker, and namespace, hivemq, with the Hashicorp Vault policy, hivemq. The tokens returned after authentication are valid for 24 hours.
Lastly, exit the vault-0 pod.
exit

Inject secrets into the pods

Switch the kubectl context to the HiveMQ Kubernetes cluster.
kubectl create namespace hivemq kubectl config set-context --current --namespace=hivemq
Install hivemq
helm upgrade oper --install hivemq/hivemq-platform-operator sleep 30 helm upgrade broker --install hivemq/hivemq-platform -n hivemq --set nodes.replicaCount=1
Check the pods in the hivemq namespace.
kubectl get pods --namespace hivemq
NAME READY STATUS RESTARTS AGE broker-0 2/2 Running 0 18m hivemq-oper-bc-vbd25 1/1 Running 0 46m
Wait until the deployed broker-0 pod reports that it is Running and ready.
Check service accounts. Copy the name of the service account of hivemq platform pods. In this example: hivemq-platform-pod-broker
kubectl get serviceaccounts
NAME SECRETS AGE default 0 4h33m hivemq-platform-operator-op 0 4h33m hivemq-platform-pod-broker 0 52m
Create a new secret required for the Hashicorp Vault authentication. vault-auth-secret.yaml:
apiVersion: v1 kind: Secret metadata: name: vault-auth-secret annotations: kubernetes.io/service-account.name: hivemq-platform-pod-broker type: kubernetes.io/service-account-token
kubectl apply --filename vault-auth-secret.yaml --namespace hivemq
Create a config map definition file configmap.yaml with a Vault Agent configuration file, vault-agent-config.hcl
apiVersion: v1 data: vault-agent-config.hcl: | # Comment this out if running as sidecar instead of initContainer exit_after_auth = true pid_file = "/home/vault/pidfile" auto_auth { method "kubernetes" { mount_path = "auth/kubernetes" config = { role = "hivemq" } } sink "file" { config = { path = "/home/vault/.vault-token" } } } template { destination = "/etc/secrets/hivemq.lic" contents = "{{ with secret (printf \"hivemq/data/test/license\") }}{{ $decoded := base64Decode .Data.data.hivemq_license_b64 }}{{ $decoded }}{{ end }}" } kind: ConfigMap metadata: name: hivemq-vault-agent-config namespace: hivemq
This creates a config map with a Vault Agent configuration file, vault-agent-config.hcl. Notice that the Vault Agent Auto-Auth (auto_auth block) is configured to use the kubernetes auth method enabled at the auth/kubernetes path on the Vault Server. The Vault Agent will use the hivemq role that you created in your Hashicorp Vault.
The sink block specifies the location on the disk where to write tokens. In this example, the sink is set to /home/vault/.vault-token.
The template block creates a file that retrieves the hivemq_license_b64 secret at the hivemq/data/test/license path, decodes it from base64, and writes the decoded text to the /etc/secrets/hivemq.lic file.
Create a ConfigMap containing a Vault Agent configuration.
kubectl create --filename configmap.yaml
Update HiveMQ Platform helm chart values by adding a new Init Container for the Vault Agent. You can get the original values by running the command
helm show values hivemq/hivemq-platform > values-hivemq-platform-with-initContainer.yaml
Add a new Init Container for the Vault Agent.
# Custom init containers to be added to the HiveMQ platform StatefulSet additionalInitContainers: - args: - agent - -config=/etc/vault/vault-agent-config.hcl - -log-level=debug env: - name: VAULT_ADDR value: http://vault.vault.svc.cluster.local:8200 image: hashicorp/vault name: vault-agent volumeMounts: - mountPath: /etc/vault name: vault-agent-config-mount - mountPath: /etc/secrets name: hivemq-license-mount
If your Hashicorp Vault is external to the HiveMQ Kubernetes cluster, replace the domain vault.vault.svc.cluster.local with the external IP of your Hashicorp Vault.
Update HiveMQ Platform helm chart values by adding the volume required by the Vault Agent Init Container configuration file and the destination volume for the HiveMQ license secret.
# Map additional volumes from possible types: "configMap", "secret", "emptyDir", "persistentVolumeClaim". additionalVolumes: - type: configMap name: hivemq-vault-agent-config mountName: vault-agent-config-mount containerName: vault-agent path: /etc/vault - type: emptyDir name: hivemq-license-volume mountName: hivemq-license-mount containerName: hivemq path: /opt/hivemq/license
Update the HiveMQ Platform
helm upgrade broker --install hivemq/hivemq-platform --values values-platform-with-initContainer.yaml

Test end-to-end

When the HiveMQ pod has started successfully the vault-agent init container will be run. The container shall authenticate to the Vault, fetch the secret, and decode the secret to the file hivemq.lic.

Check the log of the Vault Agent Init Container:
kubectl log broker-0 -c vault-agent --namespace hivemq
The following log illustrates a successful run of the Vault Agent Init Container.

Check the log of the HiveMQ container:
kubectl logs broker-0 -c hivemq --namespace hivemq | grep -i license
The following illustrates a successful use of the HiveMQ license.
INFO - Using valid site license (hivemq.lic) issued to HiveMQ GmbH for max 100 connections, valid until 2024-12-31.