Keystore as a file from Azure Key Vault | HiveMQ Platform Operator (new)

This article shows how to securely configure a TLS listener for the HiveMQ Platform on Azure Kubernetes Service (AKS) by using keystore and truststore files managed in Azure Key Vault.

• Azure CLI: You'll need the az command-line tool installed and configured with an active Azure subscription.

• kubectl: The Kubernetes command-line tool, kubectl, must be installed to interact with your AKS cluster.

• Helm: The Helm package manager for Kubernetes is required to install the HiveMQ Platform Operator.

• Keystore & Truststore: You must have your hivemq-keystore.jks and hivemq-truststore.jks files are available in your current working directory.

1. Create a resource group for the Azure Kubernetes Cluster

   az group create --name demo-group --location unitedstates

2. Create a demo AKS cluster in the resource group with Azure Key Vault Secrets Provider Add On enabled. Use the options to enable your AKS cluster to access the key vault:
   --enable-managed-identity
   --enable-addons azure-keyvault-secrets-provider

   az aks create --resource-group demo-group --name demo-cluster --node-count 2 --node-vm-size Standard_8as_v6 --kubernetes-version 1.33 --enable-managed-identity --enable-addons azure-keyvault-secrets-provider

3. Create a Key Vault in the resource group. Use the option
   --enable-rbac-authorization false
   This will enable the key vault to work with managed identities.

   az keyvault create --name demo-kv --resource-group demo-group --location unitedstates --enable-rbac-authorization false

4. Add secrets to the key vault

   az keyvault secret set --vault-name "$keyvault_name" --name keystore-password --value "changeme" az keyvault secret set --vault-name "$keyvault_name" --name keystore-private-password --value "changeme" az keyvault secret set --vault-name "$keyvault_name" --name truststore-password --value "changeme" az keyvault secret set --vault-name "$keyvault_name" --name keystore --value "$(base64 < "./tls/hivemq-broker-keystore.jks")" az keyvault secret set --vault-name "$keyvault_name" --name truststore --value "$(base64 < "./tls/hivemq-broker-truststore.jks")" az keyvault secret set --vault-name "$keyvault_name" --name license-broker --value "$(base64 < "./license/hivemq4-broker-license.lic")" az keyvault secret set --vault-name "$keyvault_name" --name license-bridge --value "$(base64 < "./license/hivemq-bridge-extension-license.elic")" az keyvault secret set --vault-name "$keyvault_name" --name license-ese --value "$(base64 < "./license/hivemq-enterprise-security-extension-license.elic")" az keyvault secret set --vault-name "$keyvault_name" --name license-kafka --value "$(base64 < "./license/hivemq-kafka-extension-license.elic")"

5. Get the Id of your Azure Key Vault Secrets Provider in the cluster. Copy the output of this command, as it is the object ID of your Azure Key Vault Secrets Provider.

   az aks show --resource-group demo-group --name demo-cluster --query addonProfiles.azureKeyvaultSecretsProvider.identity.objectId -o tsv

6. Enable your Azure Key Vault Secrets Provider to read from the key vault

   az keyvault set-policy --name demo-kv --object-id "Object Id of your Azure Key Vault Secrets Provider" --secret-permissions get list

7. Get credentials for the kubectl

   az aks get-credentials --resource-group demo-group --name demo-cluster --overwrite-existing

8. Get the Tenant Id of the key vault. Copy the output of this command, as it is the Tenant ID of your Key Vault.

   az keyvault show --name demo-kv --resource-group demo-group --query properties.tenantId -o tsv

9. Get the Client Id of the azureKeyvaultSecretsProvider in the cluster

   az aks show --resource-group "$resource_group" --name "$cluster_name" \ --query "addonProfiles.azureKeyvaultSecretsProvider.identity.clientId" \ -o tsv

10. Create SecretProviderClass manifest to bind the keystore and truststore secrets from the key vault.

    # This is a SecretProviderClass example using a user-assigned identity to access your key vault apiVersion: secrets-store.csi.x-k8s.io/v1 kind: SecretProviderClass metadata: name: demo-kv-keystores spec: provider: azure parameters: usePodIdentity: "false" useVMManagedIdentity: "true" # Set to true for using managed identity userAssignedIdentityID: "0700b0b7-3605-4d26-b447-d5e58b31267d" # The client ID of the user-assigned identity keyvaultName: "demo-kv" cloudName: "AzurePublicCloud" # [OPTIONAL for Azure] if not provided, the Azure environment defaults to AzurePublicCloud tenantId: "81cc872f-3d1e-4c97-a9a6-b6f6c6ab1b51" # The Directory ID of the key vault objects: | array: - | objectName: keystore objectType: secret objectVersion: "" objectEncoding: base64 - | objectName: truststore objectType: secret objectVersion: "" objectEncoding: base64

11. Create the SecretProviderClass

    kubectl apply -f secret-provider-class-keystores.yaml

12. Create SecretProviderClass manifest to bind the passwords secrets from the key vault to a Kubernetes Secret named hivemq-tls-passwords, :

    apiVersion: secrets-store.csi.x-k8s.io/v1 kind: SecretProviderClass metadata: name: demo-kv-passwords spec: provider: azure parameters: usePodIdentity: "false" useVMManagedIdentity: "true" # Set to true for using managed identity userAssignedIdentityID: "0700b0b7-3605-4d26-b447-d5e58b31267d" # The client ID of the user-assigned identity keyvaultName: "demo-kv" tenantId: "81cc872f-3d1e-4c97-a9a6-b6f6c6ab1b51" # The Directory ID of the key vault objects: | array: - | objectName: keystore-password objectType: secret - | objectName: keystore-private-password objectType: secret - | objectName: truststore-password objectType: secret secretObjects: - secretName: hivemq-tls-passwords type: Opaque data: - objectName: keystore-password key: keystore.password - objectName: keystore-private-password key: keystore.private.password - objectName: truststore-password key: truststore.password

13. Create the SecretProviderClass

    kubectl apply -f secrets-store-class-tls-passwords.yaml

14. Create SecretProviderClass manifest to bind the license secrets from the key vault,

    # This is a SecretProviderClass example using a user-assigned identity to access your key vault apiVersion: secrets-store.csi.x-k8s.io/v1 kind: SecretProviderClass metadata: name: demo-kv-hivemq-licenses spec: provider: azure parameters: usePodIdentity: "false" useVMManagedIdentity: "true" # Set to true for using managed identity userAssignedIdentityID: "087a6213-c25d-4493-92a6-492c90c2c612" # The client ID of the user-assigned identity keyvaultName: "demo-kv-hivemq" cloudName: "AzurePublicCloud" # [OPTIONAL for Azure] if not provided, the Azure environment defaults to AzurePublicCloud tenantId: "81cc872f-3d1e-4c97-a9a6-b6f6c6ab1b51" # The Directory ID of the key vault objects: | array: - | objectName: license-broker objectAlias: hivemq4.lic objectType: secret objectVersion: "" objectEncoding: base64 - | objectName: license-bridge objectAlias: hivemq-bridge.elic objectType: secret objectVersion: "" objectEncoding: base64 - | objectName: license-ese objectAlias: hivemq-ese.elic objectType: secret objectVersion: "" objectEncoding: base64 - | objectName: license-kafka objectAlias: hivemq-kafka.elic objectType: secret objectVersion: "" objectEncoding: base64

15. Run a test pod to preview the resulting secrets from the key vault

    kubectl apply -f pod.yaml

    Monitor for errors with:

    kubectl events

16. Install hivemq-platform-operator

    helm repo add hivemq https://hivemq.github.io/helm-charts helm repo update hivemq helm install hpo hivemq/hivemq-platform-operator

    Here we use “hpo” as a Helm release name for the hivemq-platform-operator. You can use your own naming.

17. Update the values.yaml of your hivemq-platform deployment, where you update the secure MQTT service configuration with keystore and trustore and their passwords, as well as license:

    # Configures all HiveMQ licenses. license: create: false name: "hivemq-licences" services: # Secure MQTT service configuration - type: mqtt exposed: true containerPort: 8883 keystoreSecretName: "hivemq-tls-keystores" keystoreSecretKey: "keystore" keystorePassword: "" keystorePrivatePassword: "" keystorePasswordSecretName: "hivemq-tls-passwords" keystorePasswordSecretKey: "" keystorePrivatePasswordSecretKey: "keystore.private.password" truststoreSecretName: "hivemq-tls-keystores" truststoreSecretKey: "truststore" truststorePassword: "" truststorePasswordSecretName: "hivemq-tls-passwords" truststorePasswordSecretKey: "" tlsClientAuthenticationMode: "REQUIRED"

18. Template the future hivemq-platform deployment manifests

    helm template hp hivemq/hivemq-platform --values your-values.yaml --namespace hivemq --output-dir manifests1

19. Inspect the generated manifests and find the Custom Resource Definition in the hivemq-custom-resource.yml:

    manifests1 └── hivemq-platform └── templates ├── hivemq-configuration.yml ├── hivemq-custom-resource.yml

20. Inspect the hivemq-custom-resource.yml, which will contain the specification of the stateful set

    yq '.spec.statefulSet' < manifests1/hivemq-platform/templates/hivemq-custom-resource.yml > stateful-set.yaml

21. Within the Stateful Set manifest, find the volumes: section and identify the volume, which is expected to contain the HiveMQ TLS Keystore and Truststore:

    Open

    

22. Update the volume not to use a Secret as a source, but a CSI secret from the vault instead

    spec: template: spec: containers: - name: hivemq ... volumeMounts: - name: hivemq-tls-keystores mountPath: /tls-hivemq-tls-keystores readOnly: true volumes: - name: hivemq-tls-keystores csi: driver: secrets-store.csi.k8s.io readOnly: true volumeAttributes: secretProviderClass: "demo-kv-keystores"

    Here, demo-kv-keystores is the name of the SecretProviderClass that binds the keystore and truststore secrets from the key vault to Kubernetes.

23. Within the Stateful Set manifest, find the volumes: section and identify the volume, which is expected to contain the HiveMQ licenses:

    Open

    

     

24. Update the licenses volume not to use a Secret as a source, but a CSI secret from the vault instead

    volumeMounts: - name: licenses mountPath: /opt/hivemq/license - name: hivemq-tls-keystores mountPath: /tls-hivemq-tls-keystores readOnly: true volumes: - name: licenses csi: driver: secrets-store.csi.k8s.io readOnly: true volumeAttributes: secretProviderClass: demo-kv-hivemq-licenses - name: hivemq-tls-keystores csi: driver: secrets-store.csi.k8s.io readOnly: true volumeAttributes: secretProviderClass: demo-kv-hivemq-keystores

25. Generate manifests with override of the Stateful Set to verify that everything looks correct

    helm template hp hivemq/hivemq-platform --values your-values.yaml --set-file config.overrideStatefulSet=stateful-set.yaml --output-dir manifests2
    manifests2 └── hivemq-platform └── templates ├── hivemq-configuration.yml ├── hivemq-custom-resource.yml

26. Install hivemq-platform override the stateful set:

    helm install hp hivemq/hivemq-platform --values your-values.yaml --set-file config.overrideStatefulSet=stateful-set.yaml

    Here we use “hp" as hivemq-platform’s release name. You can use your own naming for hivemq deployments.

27. First, monitor Kubernetes Events, which will first indicate any issue with access to the key vault.

    kubectl events

28. Monitor the operator log. After deploying hivemq-platform it is useful to monitor the hivemq-platform-operator log to see if there are errors during the deployment of the new hivemq-platform. For example, it might indicate that secrets are not found or a volume mount has failed.

    kubectl logs deployment/hivemq-hpo -f

29. Monitor the HiveMQ log. If the hivemq-platform pods have started, it is useful to monitor the hivemq-platform logs for possible errors. For example, to verify that the TLS listener has started successfully.

    kubectl logs statefulset/hp --follow

30. Clean up after the demo. Delete the AKS cluster.

    az aks delete --resource-group demo-group --name demo-cluster

31. Delete the kubectl profile

    kubectl config delete-context demo-cluster

32. Delete the Key Vault

    az keyvault delete --name demo-kv --resource-group demo-group

33. Purge the Key Vault

    az keyvault purge --name demo-kv --location unitedstates

34. Delete the resource group

    az group delete --name demo-group