External Secrets from Hashicorp Vault | HiveMQ Platform Operator (new)

This article explains how to fetch secrets from an external Hashicorp Vault and put them into Kubernetes secrets required for the HiveMQ Platform, such as license, keystore, and truststore.

\uD83D\uDCD8 Instructions

First, ensure you have the VAULT_ADDR and VAULT_TOKEN from the external vault at hand.

Next, put your secrets in the vault. In this article, we store files in the vault encoded with base64.

putting files and strings to the vault

The files (license, keystore, truststore):

$ ls -1 /tmp
broker-keystore.jks
broker-truststore.jks
hivemq.lic

The files from the /tmp directory we put to the vault:

vault secrets enable -path=hivemq-mqtt/obc-poc kv-v2
cd /tmp
vault kv put hivemq-mqtt/obc-poc/opt/hivemq/conf/broker-keystore keystore=\"\$(base64 -w 0 < broker-keystore.jks)\" \
  keystore.password=changeme keystore.passphrase=changeme
vault kv put hivemq-mqtt/obc-poc/opt/hivemq/conf/broker-truststore truststore=\"\$(base64 -w 0 < broker-truststore.jks)\" \
  truststore.password=changeme
vault kv put hivemq-mqtt/obc-poc/opt/hivemq/license \"latest=\$(base64 -w 0 hivemq.lic)\"
vault auth enable kubernetes
vault write auth/kubernetes/config \"kubernetes_host=https://\${KUBERNETES_PORT_443_TCP_ADDR}:443\"
vault policy write hivemq - <<EOF
path \"hivemq-mqtt/obc-poc/data/opt/hivemq/license\" {
   capabilities = [\"read\"]
}
path \"hivemq-mqtt/obc-poc/data/opt/hivemq/conf\" {
   capabilities = [\"read\"]
}
EOF

Install External Secrets Kubernetes Operator

Install external-secrets/external-secrets in the same namespace as HiveMQ Platform.

helm install external-secrets external-secrets/external-secrets --namespace hivemq

Create the secret required for the external-secrets to access the external vault. The secret should contain the VAULT_TOKEN. In our case, the token is root

kubectl create secret generic secret-token-for-vault --from-literal token=root  --namespace hivemq

OR

kubectl apply -f 1-secret-token-for-vault.yaml --namespace hivemq

apiVersion: v1
kind: Secret
metadata:
  name: secret-token-for-vault
data:
  token: cm9vdA== # "root"

Create the Secret Store required for the external-secrets to access the external vault. The definition should contain the vault URL and reference to the secret with the token, and also the path (path in the kv-2 engine in the external vault from which the token is allowed to read)

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: secretstore-vault
spec:
  provider:
    vault:
      server: "http://[ VAULT_ADDR ]:8200"
      path: "hivemq-mqtt/obc-poc"
      version: "v2"
      auth:
        # points to a secret that contains a vault token
        # https://www.vaultproject.io/docs/auth/token
        tokenSecretRef:
          name: "secret-token-for-vault"
          key: "token"

kubectl apply -f 2-secretstore.yaml --namespace hivemq

Now the external-secrets should be able to access the external vault successfully

Create an external-secret that will fetch the keystore from the external vault and put it to the Kubernetes secret hivemq-keystore-v, key keystore.

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: external-secret-for-keystore
spec:
  refreshInterval: "15s"
  secretStoreRef:
    name: secretstore-vault
    kind: SecretStore
  data:
    - remoteRef:
        decodingStrategy: Base64
        key: "hivemq-mqtt/obc-poc/opt/hivemq/conf/broker-keystore"
        property: keystore
      secretKey: keystore
  target:
    name: hivemq-keystore-v

Apply the rest of the external secrets manifests in the same fashion:

If everything works, the following Kubernetes Secrets will be created automatically:

kubectl get secrets --namespace hivemq

hivemq-keystore-passphrase-v             Opaque               1      54m
hivemq-keystore-password-v               Opaque               1      54m
hivemq-keystore-v                        Opaque               1      54m
hivemq-license-v                         Opaque               1      24m
hivemq-truststore-password-v             Opaque               1      54m
hivemq-truststore-v                      Opaque               1      54m

Install HiveMQ Platform

Now, update the HiveMQ Platform values.yaml and configure that license, keystore, trustore, and their passwords are taken from relevant secrets.
Specify that the license should be taken from the secret hivemq-license-v

# Configures the HiveMQ License information.
license:
  create: false
  name: "hivemq-license-v"
  data: ""
  overrideLicense: ""

Specify that the keystore should be taken from the secret hivemq-keystore-v

  # Secure WebSocket service configuration
  - type: websocket
    exposed: true
    containerPort: 8000
    
    keystoreSecretName: "hivemq-keystore-v"
    keystoreSecretKey: "keystore"

    keystorePassword: ""
    keystorePrivatePassword: ""

    keystorePasswordSecretName: "hivemq-keystore-password-v"
    keystorePasswordSecretKey: "keystore.password"
    keystorePrivatePasswordSecretKey: "keystore.password"
    
    # Mutual TLS configuration values
    truststoreSecretName: "hivemq-truststore-v"
    truststoreSecretKey: "truststore"
    truststorePassword: ""
    truststorePasswordSecretName: "hivemq-truststore-password-v"
    tlsClientAuthenticationMode: "OPTIONAL"

Example HiveMQ Platform values.yaml:

Install the HiveMQ Platform Operator and HiveMQ Platform

helm upgrade op --install hivemq/hivemq-platform-operator --set logLevel=DEBUG --namespace hivemq

helm upgrade op --install hivemq/hivemq-platform --values values-hivemq-platform.yaml --namespace hivemq

Check the HiveMQ broker stateful set logs to make sure the license, keystore and trustore are applied correctly.

External Secrets from Hashicorp Vault | HiveMQ Platform Operator (new)

\uD83D\uDCD8 Instructions

Install External Secrets Kubernetes Operator

Install HiveMQ Platform

\uD83D\uDCCB Related articles