Table of Contents |
---|
...
HiveMQ Enterprise Extension For Kafka requires a separate license file, e.g. kafka-license.elic, in the $HIVEMQ_HOME/license directory. You can skip this step. If you skip this step, then the kafka-extension will start in trial mode, limited to 5h, and will be automatically disabled by the HiveMQ broker after 5h. To add the kafka-license.elic along with the hivemq-license.lic, create a new configmap hivemq-license including all desired license files:
Code Block language bash kubectl create configmap hivemq-license --namespace=hivemq \ --from-file hivemq-license.lic \ --from-file kafka-license.elic
In case you get error
configmaps "hivemq-license" already exists , please delete the last configmap using kubectl delete configmap hivemq-license --namespace hivemq and try the addition step again
.Edit the values.yaml file of the hivemq-operator, section
hivemq.configMaps
. Update this:Code Block configMaps: [] # ConfigMaps to mount to the HiveMQ pods. These can be mounted to existing directories without shadowing the folder contents as well. #- name: hivemq-license # path: /opt/hivemq/license
To this:
Code Block configMaps: - name: hivemq-license path: /opt/hivemq/license
This will mount the content of the configMap
hivemq-license
to the directory/opt/hivemq/license
of the hivemq-broker pods.
------ Configuring Kafka Extension-----------
HiveMQ Enterprise Extension For Kafka is preinstalled with HiveMQ so once you enable it, it will look for its configuration file. You must prepare this file before enabling the extension. If you skip this step, the extension will not find its configuration file and will not load any configuration.Prepare a simple configuration file for kafka-extension as in the example below.
this example configuration will map all incoming MQTT publish packets to the topic “test” in Kafka; and will map the topic “test” in Kafka to the topic “test” in the HiveMQ broker
Use your password in
<password>here_is_your_password</password>
, that you successfully retrieved with this command a few steps ago:Code Block language bash kubectl get secret kafka-user-passwords --namespace kafka \ -o jsonpath='{.data.client-passwords}' | base64 -d | cut -d , -f 1;
Here is the file:
...
Create a new configMap
kafka-config
from theconfig.xml
:Code Block language bash kubectl create configmap kafka-config --from-file config.xml --namespace hivemq
Edit values.yaml file of
hivemq-operator
and update section hivemq.extensions, havingname: hivemq-kafka-extension
Update this:Code Block language yaml extensions: - name: hivemq-kafka-extension extensionUri: preinstalled enabled: false
To this:
Code Block language yaml extensions: - name: hivemq-kafka-extension extensionUri: preinstalled enabled: true configMap: kafka-config initialization: | # Fixes the location of the config.xml file [[ ! -f conf/config.xml ]] && [[ -f /conf-override/extensions/hivemq-kafka-extension/config.xml ]] && ln -s /conf-override/extensions/hivemq-kafka-extension/config.xml conf/config.xml
Re-deploy hivemq-operator with updated values.yaml
Code Block language bash helm upgrade hivemq --install hivemq/hivemq-operator --values values.yaml --namespace hivemq
– for ease of use we switch namespace back to hivemq
kubectl config set-context --current --namespace=hivemq
If everything is correct,
The HiveMQ log contains info about using the correct license:
Code Block language bash kubectl logs deployment/hivemq | grep 'Using valid'
Code Block language text INFO - Using valid Enterprise Edition CPU license (hivemq-license.lic) issued to HiveMQ - Internal for max 9999 CPU cores, valid until 2024-03-31. INFO - Using valid license (kafka-license.elic) for enterprise extension with name "HiveMQ Enterprise Extension for Kafka", valid until 2024-03-31.
The HiveMQ log contains info about successful Kafka connection:
Code Block language bash kubectl logs deployments/hivemq -f | grep -i kafka
Code Block language text INFO - Starting extension with id "hivemq-kafka-extension" at /opt/hivemq/extensions/hivemq-kafka-extension INFO - No mqtt-to-kafka transformers were found. No mqtt-to-kafka transformers are started. INFO - Started mqtt-to-kafka mapping "mapping01" with MQTT topics [#] and Kafka topic "test" for Kafka Cluster "cluster01" INFO - No kafka-to-mqtt transformers were found. No kafka kafka-to-mqtt transformers are started. INFO - Started kafka-to-mqtt mapping "mapping02" with kafka topic: [test] and kafka pattern: [test-(.)*] for Kafka Cluster "cluster01" (poll duration: 100 ms). INFO - Extension "HiveMQ Enterprise Extension for Kafka" version 4.24.0 started successfully. INFO - Subscribing kafka-to-mqtt topic-mapping with id mapping02 to kafka pattern (test-(.)*|\Qtest\E)
Kafka dashboard is visible in the HiveMQ Control Center:
...
Subscribe a reference mqtt client to the topic “test”, output show the topic name and message: (please update your hostname with the DNS name or public IP address of your hivemq-hivemq-mqtt service, 20.113.46.120 in our test)
Code Block language bash mqtt subscribe --topic '#' --host $hivemqhost20.113.46.120 --port 1883 -q 1 --showTopics
Do not close this terminal session!
From a different terminal session, publish a message to the topic “test”:(please update your hostname with the DNS name or public IP address of your hivemq-hivemq-mqtt service)
Code Block language bash mqtt publish --topic test --message Hello --host $hivemqhost20.113.46.120 --port 1883 -q 1
If everything is correct, the subscriber will indefinitely receive the message we published:
Code Block mqtt subscribe --topic '#' --host $hivemqhost --port 1883 -q 1 --showTopics test: Hello test: Hello test: Hello test: Hello test: Hello test: Hello test: Hello test: Hello test: Hello
You can stop this by terminating the execution of the command by pressing Ctrl+C.
To gain extra points for this training, explain, why the message is received indefinitely rather than once.
If everything is correct, the Kafka Dashboard in the HiveMQ Control Center shows incoming and outgoing Kafka messaging:
...