Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

In this knowledge base article, we set up Hashicorp Vault and injector service with the Hashicorp Vault Helm chart and store a HiveMQ License as a secret. Then we will deploy the HiveMQ broker cluster with the hivemq-operator Helm chart to demonstrate how the Hashicorp Vault injector service retrieves, decodes and writes the secret to /opt/hivemq/license/hivmq.lic file on the pod for the HiveMQ application to use.

Prerequisites

These instructions require the following tools on the local machine:

...

The kubectl context should be set to the Kubernetes cluster where the HiveMQ broker will be installed.

\uD83D\uDCD8 Instructions

Install the Hashicorp Vault Helm Chart

  1. Add the HashiCorp Helm repository.

    Code Block
    languagebash
    helm repo add hashicorp https://helm.releases.hashicorp.com
  2. Update all the repositories to ensure helm is aware of the latest versions.

    Code Block
    languagebash
    helm repo update hashicorp
  3. Install the latest version of the Hashicorp Vault server running in development mode.
    Development mode: Running a Hashicorp Vault server in development is automatically initialized and unsealed. This is ideal in a learning environment but NOT recommended for a production environment.

    Code Block
    languagebash
    helm install vault hashicorp/vault --set "server.dev.enabled=true"

    The vault pod and vault Agent Injector pod are deployed in the default namespace.

  4. Display all the pods in the default namespace.

    Code Block
    languagebash
    kubectl get pods
    Code Block
    languagetext
    NAME                                    READY   STATUS    RESTARTS   AGE
    vault-0                                 1/1     Running   0          80s
    vault-agent-injector-5945fb98b5-tpglz   1/1     Running   0          80s

    The vault-0 pod runs a vault server in development mode. The vault-agent-injector pod performs the injection based on the annotations present or patched on a deployment.

  5. Wait until the vault-0 pod and vault-agent-injector pod are running and ready (1/1).

Set a secret in Hashicorp Vault

  1. Copy the hivemq-license file to the vault-0 pod.

    Code Block
    languagebash
    kubectl cp hivemq.lic vault-0:/tmp/
  2. Verify that the file is copied.

    Code Block
    languagebash
    kubectl exec -it vault-0 -- ls /tmp
    Code Block
    languagetext
    hivemq.lic
  3. Start an interactive shell session on the vault-0 pod.

    Code Block
    languagebash
    kubectl exec -it vault-0 -- /bin/sh

    image-20240802-143511.pngImage Modified

    Your system prompt is replaced with a new prompt / $. Commands issued at this prompt are executed on the vault-0 container.

  4. Enable kv-v2 secrets at the path hivemq.

    Code Block
    languagebash
    vault secrets enable -path=hivemq kv-v2
    Code Block
    languagetext
    Success! Enabled the kv-v2 secrets engine at: hivemq/
  5. Create a secret at path hivemq/myenv/license with a hivemq_license_b64 key and base64-encoded /tmp/hivemq.lic file.

    Code Block
    languagebash
    cd /tmp
    vault kv put hivemq/myenv/license hivemq_license_b64="$(base64 -w 0 hivemq.lic)"
    Code Block
    languagetext
    ====== Secret Path ======
    hivemq/data/myenv/license
    
    ======= Metadata =======
    Key                Value
    ---                -----
    created_time       2024-02-21T17:34:39.261249639Z
    custom_metadata    <nil>
    deletion_time      n/a
    destroyed          false
    version            1
  6. Verify that the secret is defined at the path hivemq/myenv/license.

    Code Block
    languagebash
    vault kv get hivemq/myenv/license
    Code Block
    languagetext
    ====== Secret Path ======
    hivemq/data/myenv/license
    
    ======= Metadata =======
    Key                Value
    ---                -----
    created_time       2024-02-21T14:57:01.446984026Z
    custom_metadata    <nil>
    deletion_time      n/a
    destroyed          false
    version            1
    
    ========= Data =========
    Key                 Value
    ---                 -----
    hivemq_license_b64  SCFNUSRbM10.......
    

    The secret is ready for the application.

  7. Lastly, exit the vault-0 pod.

    Code Block
    languagebash
    exit

Configure Kubernetes authentication

Hashicorp Vault provides a Kubernetes authentication method that enables clients to authenticate with a Kubernetes Service Account Token. This token is provided to each pod when it is created.

  1. Start an interactive shell session on the vault-0 pod.

    Code Block
    languagebash
    $ kubectl exec -it vault-0 -- /bin/sh
    
    / $

    image-20240802-170613.pngImage Added

    Your system prompt is replaced with a new prompt / $. Commands issued at this prompt are executed on the vault-0 container.

  2. Enable the Kubernetes authentication method.

    Code Block
    languagebash
    $ vault auth enable kubernetes
    
    
    Code Block
    languagetext
    Success! Enabled kubernetes auth method at: kubernetes/

    Hashicorp Vault accepts a service token from any client in the Kubernetes cluster. During authentication, Hashicorp Vault verifies that the service account token is valid by querying a token review Kubernetes endpoint.

  3. Configure the Kubernetes authentication method to use the location of the Kubernetes API.

    Note: For the best compatibility with recent Kubernetes versions, ensure you are using Hashicorp Vault v1.13.3 or greater.

    Code Block
    languagebash
    $ vault write auth/kubernetes/config \
          kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443"

    Successful output from the command resembles this example:

    Code Block
    languagetext
    Success! Data written to: auth/kubernetes/config

    The environment variable KUBERNETES_PORT_443_TCP_ADDR is defined and references the internal network address of the Kubernetes host.

  4. For a client to read the secret data defined at hivemq/myenv/license, requires that the read capability be granted for the path hivemq/data/myenv/license. This is an example of a policy. A policy defines a set of capabilities.

  5. Write out the policy named hivemq that enables the read capability for secrets at path hivemq/data/myenv/license.

    Code Block
    languagebash
    $ vault policy write hivemq - <<EOF
    path "hivemq/data/myenv/license" {
       capabilities = ["read"]
    }
    EOF
  6. Create a Kubernetes authentication role named hivemq.

    Code Block
    languagebash
    $ vault write auth/kubernetes/role/hivemq \
          bound_service_account_names=hivemq-hivemq-operator-hivemq \
          bound_service_account_namespaces=hivemq \
          policies=hivemq \
          ttl=24h

    Successful output from the command resembles this example:

    Code Block
    languagetext
    Success! Data written to: auth/kubernetes/role/hivemq

    The role connects the Kubernetes service account, hivemq-hivemq-operator-hivemq, and namespace, hivemq, with the Hashicorp Vault policy, hivemq. The tokens returned after authentication are valid for 24 hours.

  7. Lastly, exit the vault-0 pod.

    Code Block
    languagebash
    $ exit

Inject secrets into the pods

  1. If you do not have values.yaml file yet, you can get the latest version from the Helm chart repository and store it as a file, for example, values-hivemq.yaml:

    Code Block
    languagebash
    helm show values hivemq/hivemq-operator > values-hivemq.yaml
  2. Edit the values-hivemq.yaml file. Add annotations to the hivemq pods.

    Code Block
    languageyaml
    hivemq:
      # Annotations to add to the HiveMQ Pods
      podAnnotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/role: "hivemq"
        vault.hashicorp.com/agent-inject-status: 'update'
        vault.hashicorp.com/agent-inject-secret-hivemq.lic: "hivemq/data/myenv/license"
        vault.hashicorp.com/secret-volume-path-hivemq.lic: "/opt/hivemq/license/"
        vault.hashicorp.com/agent-inject-template-hivemq.lic: |
          {{- with secret \"hivemq/data/myenv/license\" -}}
          {{- $hivemq_broker_license := base64Decode .Data.data.hivemq_license_b64 -}}
          {{- $hivemq_broker_license -}}
          {{- end -}}
  3. (Re)install hivemq

    Code Block
    languagebash
    helm upgrade hivemq --install hivemq/hivemq-operator -n hivemq -f values-hivemq.yaml
  4. Get all the pods in the hivemq namespace.

    Code Block
    languagebash
    $ kubectl get pods --nnamespace hivemq
    
    Code Block
    languagetext
    NAME                                    READY   STATUS     RESTARTS   AGE
    hivemq-599cb74d9c-s8hhm                 0/2     Init:0/1   0          23s
    hivemq-69697d9598-l878s                 1/1     Running    0          20m
    vault-0                                 1/1     Running    0          78m
    vault-agent-injector-5945fb98b5-tpglz   1/1     Running    0          78m

    Wait until the re-deployed hivemq pod reports that it is Running and ready (2/2).

    This new pod now launches two containers. The application container, named hivemq, and the Hashicorp Vault Agent container, named vault-agent.

  5. Display the logs of the vault-agent container in the new hivemq pod.

    Code Block
    languagebash
    $ kubectl logs \
          $(kubectl get pod -l app=hivemq -o jsonpath="{.items[0].metadata.name}") \
          --container vault-agent

    Hashicorp Vault Agent manages the token lifecycle and the secret retrieval. The secret is rendered in the hivemq container at the path /opt/hivemq/license/.

  6. Display the secret written to the hivemq container.

    Code Block
    languagebash
    $ kubectl exec \
          $(kubectl get pod -l app=hivemq -o jsonpath="{.items[0].metadata.name}") \
          --container hivemq -- cat /opt/hivemq/license/hivemq.lic

    The base64-decoded secret data is present on the container (smile)

https://developer.hashicorp.com/vault/tutorials/kubernetes/kubernetes-sidecar?ajs_aid=f51d5b2f-f5e5-4e88-8689-d479a67a2ae8&product_intent=vault#inject-secrets-into-the-pod

...