Set up Kafka extension using HiveMQ operator

Install Kafka using helm

1. Create a namespace for Kafka and switch the context to it:

   kubectl create namespace kafka;
   kubectl config set-context --current --namespace=kafka

2. Add the repository for the Kafka Helm chart to your package manager.

   helm repo add bitnami https://charts.bitnami.com/bitnami

3. Deploy the Kafka server using the Helm chart.

   1. Additional step with AWS EKS cluster:
      You will need to configure the Amazon EBS CSI driver. You can find detailed steps How to configure Amazon EBS CSI driver for working PersistentVolumes in EKS

   2. The below command deploys Kafka with 2 brokers (replicas).

4. Please notice the output of the command above, it provides critical data that is used for the next steps

   1. Consumers can access Kafka via port 9092 on the following DNS name from within your cluster: kafka.kafka.svc.cluster.local

   2. The CLIENT listener for Kafka client connections from within your cluster has been configured with the following security settings: SASL authentication

   3. To connect a client to your Kafka:

      1. username="user1"

      2. To get the password execute the command below:(skip % at the end)

Configuring the Kafka Extension

Setting up the Kafka license as a ConfigMap

1. HiveMQ Enterprise Extension For Kafka requires a separate license file, e.g. kafka-license.elic, in the $HIVEMQ_HOME/license directory. To add the kafka-license.elic along with the hivemq-license.lic, create a new configmap hivemq-license including all desired license files:

2. Edit the values.yaml file of the hivemq-operator, section hivemq.configMaps. Update this:

   To this:

   This will mount the content of the configMap hivemq-license to the directory /opt/hivemq/license of the hivemq-broker pods.

Configuring the extension

1. Prepare a simple configuration file for kafka-extension as in the example below.

   • this example configuration will map all incoming MQTT publish packets to the topic “test” in Kafka; and will map the topic “test” in Kafka to the topic “test” in the HiveMQ broker

   • Use your password in <password>here_is_your_password</password>, that you successfully retrieved with this command a few steps ago:

   • Here is the file:

2. Create a new configMap kafka-config from the config.xml:

3. Edit values.yaml file of hivemq-operator and update section hivemq.extensions, having name: hivemq-kafka-extension
   Update this:

   To this:

4. Re-deploy hivemq-operator with updated values.yaml

   – for ease of use we switch namespace back to hivemq kubectl config set-context --current --namespace=hivemq

Check if the license and configuration is applied correctly

If everything is correct,

1. The HiveMQ log contains info about using the correct license:

2. The HiveMQ log contains info about successful Kafka connection:

3. Kafka dashboard is visible in the HiveMQ Control Center:

Open

Testing Message Flow between HiveMQ Broker and Kafka using MQTT CLI

1. Subscribe a reference MQTT client to the topic “test”, The output shows the topic name and message: (please update your hostname with the DNS name or public IP address of your hivemq-hivemq-mqtt service, 20.113.46.120 in our test)

   Do not close this terminal session!

2. From a different terminal session, publish a message to the topic “test”:(please update your hostname with the DNS name or public IP address of your hivemq-hivemq-mqtt service, 20.113.46.120 in our test)

3. If everything is correct, the subscriber will indefinitely receive the message we published. You can stop this by terminating the execution of the command by pressing Ctrl+C.

4. the Kafka Dashboard in the HiveMQ Control Center shows incoming and outgoing Kafka messaging:

Open