Setting Up Enterprise Kafka Extension with HiveMQ Platform Operator

This guide provides detailed steps for configuring the Enterprise Kafka Extension with the HiveMQ Platform Operator. Ensure you meet the specified prerequisites before proceeding.

Prerequisites:

1. Helm version v3+

2. Running Kubernetes cluster version 1.18.0 or higher

3. kubectl latest version

 Instructions

1. Setup Kafka cluster: (This step is optional if you already have a running Kafka cluster).

   1. Create a namespace for Kafka and switch the context to it:

      kubectl create namespace kafka;
      kubectl config set-context --current --namespace=kafka

   2. Add the repository for the Kafka Helm chart to your package manager.

      helm repo add bitnami https://charts.bitnami.com/bitnami
      helm repo update

   3. Deploy the Kafka server using the Helm chart.

      1. The below command deploys Kafka with 2 brokers (replicas).

         helm upgrade --install kafka bitnami/kafka --namespace=kafka --set replicaCount=2

   4. Please notice the output of the command above, it provides critical data that is used for the next steps

      1. Consumers can access Kafka via port 9092 on the following DNS name from within your cluster: kafka.kafka.svc.cluster.local

      2. The CLIENT listener for Kafka client connections from within your cluster has been configured with the following security settings: SASL authentication

      3. To connect a client to your Kafka:

         1. username="user1"

         2. To get the password execute the command below:(skip % at the end)

            kubectl get secret kafka-user-passwords --namespace kafka \ -o jsonpath='{.data.client-passwords}' | base64 -d | cut -d , -f 1;

2. Deploy HiveMQ with Kafka extension

   1. Create a namespace for HiveMQ installation and switch the context to it:

      kubectl create namespace hivemq;
      kubectl config set-context --current --namespace=hivemq

   2. Generate hivemq_values.yaml:

      Deploy HiveMQ using the HiveMQ Platform and generate the hivemq_values.yaml file:

      helm show values hivemq/hivemq-platform > hivemq_values.yaml

   3. Configure Kafka extension License:

      Follow the specific steps outlined for configuring the HiveMQ and Kafka extension license. Setting Up HiveMQ License for Your HiveMQ Cluster using HiveMQ Platform Operator

   4. Create config.xml for Kafka extension:

      1. Examples of the config.xml file are in the extension folder under conf/examples.

      2. Configure the username and password for successful authentication to your Kafka cluster.

         1. If you have followed our guide to deploy Kafka cluster then use the username as “user1” and password you copied from the step.

      3. Configure the <mqtt-to-kafka-mappings> or <kafka-to-mqtt-mappings> or <kafka-to-mqtt-transformers> or <mqtt-to-kafka-transformers> based on your need.

      4. Please refer to the example:

         <?xml version="1.0" encoding="UTF-8" ?> <kafka-configuration xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="config.xsd"> <kafka-clusters> <kafka-cluster> <id>cluster01</id> <bootstrap-servers>kafka.kafka.svc.cluster.local:9092</bootstrap-servers> <authentication> <scram-sha256> <username>user1</username> <password>here_is_your_password</password> </scram-sha256> </authentication> </kafka-cluster> </kafka-clusters>