Page Comparison

...

...

...

az login

az aks get-credentials -g hmqResourceGroup -n HiveMQCluster

Install Kafka using helm

Create a namespace for Kafka and switch the context to it:

...

Install Kafka using helm

1. Create a namespace for Kafka and switch the context to it:

   Code Block
   language bash
   kubectl create namespace kafka;

   Code Block
   kubectl config set-context --current --namespace=kafka

2. Add the repository for the Kafka Helm chart to your package manager.

   Code Block
   language bash
   helm repo add bitnami https://charts.bitnami.com/bitnami

   Code Block
   language bash
   helm repo update bitnami

3. Deploy the Kafka server using the Helm chart. The below command deploys Kafka with 2 brokers (

   1. Additional step with AWS EKS cluster:
      You will need to configure the Amazon EBS CSI driver. You can find detailed steps How to configure Amazon EBS CSI driver for working PersistentVolumes in EKS

   2. The below command deploys Kafka with 2 brokers (replicas).

      Code Block
      language bash
      helm upgrade --install kafka bitnami/kafka --namespace=kafka --set replicaCount=2

      If everything is correct, then

      Kafka can be accessed by consumers

4. Please notice the output of the command above, it provides critical data that is used for the next steps

   1. Consumers can access Kafka via port 9092 on the following DNS name from within your cluster:

      kafka.kafka.svc.cluster.local

   2. The CLIENT listener for Kafka client connections from within your cluster has been configured with the following security settings:

      SASL authentication

   3. To connect a client to your Kafka:

      1. username="user1"

      2. To get the password execute the command below:(skip % at the end)

         Code Block
         language bash
         kubectl get secret kafka-user-passwords --namespace kafka \ -o jsonpath='{.data.client-passwords}' | base64 -d | cut -d , -f 1;

...

Configuring the Kafka Extension

Setting up the Kafka license as a ConfigMap

1. HiveMQ Enterprise Extension For Kafka requires a separate license file, e.g. kafka-license.elic, in the $HIVEMQ_HOME/license directory. You can skip this step. If you skip this step, then the kafka-extension will start in trial mode, limited to 5h, and will be automatically disabled by the HiveMQ broker after 5hthe $HIVEMQ_HOME/license directory. To add the kafka-license.elic along with the hivemq-license.lic, create a new configmap hivemq-license including all desired license files:

   Code Block
   language bash
   kubectl create configmap hivemq-license --namespace=hivemq \ --from-file hivemq-license.lic \ --from-file kafka-license.elic

   In case you get error configmaps "hivemq-license" already exists , please delete the last configmap using kubectl delete configmap hivemq-license --namespace hivemq and try the addition step again.

   --from-file hivemq-license.lic \ --from-file kafka-license.elic

2. Edit the values.yaml file of the hivemq-operator, section hivemq.configMaps. Update this:

   Code Block
   configMaps: [] # ConfigMaps to mount to the HiveMQ pods. These can be mounted to existing directories without shadowing the folder contents as well. #- name: hivemq-license # path: /opt/hivemq/license

   To this:

   Code Block
   configMaps: - name: hivemq-license path: /opt/hivemq/license

   This will mount the content of the configMap hivemq-license to the directory /opt/hivemq/license of the hivemq-broker pods.
   ------ Configuring Kafka Extension-----------hivemq-broker pods.

Configuring the extension

1. Prepare a simple configuration file for kafka-extension as in the example below.

   • this example configuration will map all incoming MQTT publish packets to the topic “test” in Kafka; and will map the topic “test” in Kafka to the topic “test” in the HiveMQ broker

   • Use your password in <password>here_is_your_password</password>, that you successfully retrieved with this command a few steps ago:

     Code Block
     language bash
     kubectl get secret kafka-user-passwords --namespace kafka \ -o jsonpath='{.data.client-passwords}' | base64 -d | cut -d , -f 1;

   • Here is the file:

<?xml version="1.0" encoding="UTF-8" ?>
<kafka-configuration xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                     xsi:noNamespaceSchemaLocation="config.xsd">
    <kafka-clusters>
        <kafka-cluster>
            <id>cluster01</id>
            <bootstrap-servers>kafka.kafka.svc.cluster.local:9092</bootstrap-servers>
            <authentication>
                <scram-sha256>
                    <username>user1</username>
                    <password>here_is_your_password</password>
                </scram-sha256>
            </authentication>
        </kafka-cluster>
    </kafka-clusters>

    <mqtt-to-kafka-mappings>
        <mqtt-to-kafka-mapping>
            <id>mapping01</id>
            <cluster-id>cluster01</cluster-id>
            <mqtt-topic-filters>
                <mqtt-topic-filter>#</mqtt-topic-filter>
            </mqtt-topic-filters>
            <kafka-topic>test</kafka-topic>
        </mqtt-to-kafka-mapping>
    </mqtt-to-kafka-mappings>

    <kafka-to-mqtt-mappings>
        <kafka-to-mqtt-mapping>
            <id>mapping02</id>
            <cluster-id>cluster01</cluster-id>
            <kafka-topics>
                <kafka-topic>test</kafka-topic>
                <kafka-topic-pattern>test-(.)*</kafka-topic-pattern>
            </kafka-topics>
        </kafka-to-mqtt-mapping>
    </kafka-to-mqtt-mappings>
</kafka-configuration>

2. Create a new configMap kafka-config from the config.xml:

kubectl create configmap kafka-config --from-file config.xml --namespace hivemq

3. Edit values.yaml file of hivemq-operator and update section hivemq.extensions, having name: hivemq-kafka-extension
   Update this:

   Code Block
   language yaml
   extensions: - name: hivemq-kafka-extension extensionUri: preinstalled enabled: false

   To this:

   Code Block
   language yaml
   extensions: - name: hivemq-kafka-extension extensionUri: preinstalled enabled: true configMap: kafka-config initialization: | # Fixes the location of the config.xml file [[ ! -f conf/config.xml ]] && [[ -f /conf-override/extensions/hivemq-kafka-extension/config.xml ]] && ln -s /conf-override/extensions/hivemq-kafka-extension/config.xml conf/config.xml

4. Re-deploy hivemq-operator with updated values.yaml

   Code Block
   language bash
   helm upgrade hivemq --install hivemq/hivemq-operator --values values.yaml --namespace hivemq

   – for ease of use we switch namespace back to hivemq kubectl config set-context --current --namespace=hivemq

Check if the license and configuration is applied correctly

If everything is correct,

1. The HiveMQ log contains info about using the correct license:

kubectl logs deployment/hivemq | grep 'Using valid'

INFO  - Using valid Enterprise Edition CPU license (hivemq-license.lic) issued to HiveMQ - Internal for max 9999 CPU cores, valid until 2024-03-31.
INFO  - Using valid license (kafka-license.elic) for enterprise extension with name "HiveMQ Enterprise Extension for Kafka", valid until 2024-03-31.

2. The HiveMQ log contains info about successful Kafka connection:

kubectl logs deployments/hivemq -f | grep -i kafka

INFO  - Starting extension with id "hivemq-kafka-extension" at /opt/hivemq/extensions/hivemq-kafka-extension
INFO  - No mqtt-to-kafka transformers were found. No mqtt-to-kafka transformers are started.
INFO  - Started mqtt-to-kafka mapping "mapping01" with MQTT topics [#] and Kafka topic "test" for Kafka Cluster "cluster01"
INFO  - No kafka-to-mqtt transformers were found. No kafka kafka-to-mqtt transformers are started.
INFO  - Started kafka-to-mqtt mapping "mapping02" with kafka topic: [test] and kafka pattern: [test-(.)*] for Kafka Cluster "cluster01" (poll duration: 100 ms).
INFO  - Extension "HiveMQ Enterprise Extension for Kafka" version 4.24.0 started successfully.
INFO  - Subscribing kafka-to-mqtt topic-mapping with id mapping02 to kafka pattern (test-(.)*|\Qtest\E)

3. Kafka dashboard is visible in the HiveMQ Control Center:

   Image Removed

...

Testing Message Flow between HiveMQ Broker and Kafka using MQTT CLI

1. Subscribe a reference mqtt MQTT client to the topic “test”, The output show shows the topic name and message: (please update your hostname with the DNS name or public IP address of your hivemq-hivemq-mqtt service, 20.113.46.120 in our test)

   Code Block
   language bash
   mqtt subscribe --topic '#' --host 20.113.46.120 --port 1883 -q 1 --showTopics

   Do not close this terminal session!

2. From a different terminal session, publish a message to the topic “test”:(please update your hostname with the DNS name or public IP address of your hivemq-hivemq-mqtt servicemqtt service, 20.113.46.120 in our test)

   Code Block
   language bash
   mqtt publish --topic test --message Hello --host 20.113.46.120 --port 1883 -q 1

3. If everything is correct, the subscriber will indefinitely receive the message we published:. You can stop this by terminating the execution of the command by pressing Ctrl+C.

   Code Block
   mqtt subscribe --topic '#' --host $hivemqhost --port 1883 -q 1 --showTopics test: Hello test: Hello test: Hello test: Hello test: Hello test: Hello test: Hello test: Hello test: Hello

   You can stop this by terminating the execution of the command by pressing Ctrl+C.

...

1. ........

2. the Kafka Dashboard in the HiveMQ Control Center shows incoming and outgoing Kafka messaging:

...