Setting Up HiveMQ Kafka Extension for Redpanda Cluster Connectivity

Owned by Sheetal Shet, created
Apr 16, 2024
3 min read

This guide aims to facilitate the setup process of the HiveMQ Kafka extension for seamless integration with a Redpanda cluster. The steps outlined below ensure a smooth configuration process, enabling efficient communication between HiveMQ and Redpanda.

Prerequisites

Before proceeding with the setup, ensure the following prerequisites are met:

  • A running Redpanda cluster. For the purpose of this guide, we utilize RedPanda’s cloud serverless cluster.

  • Authenticating user should have appropriate permissions configured on the Redpanda cluster.

  • The latest version of the HiveMQ.

 Instructions

  1. Create Kafka Config.xml:

    Begin by creating a Kafka config.xml file to establish a connection with the Redpanda cluster. Ensure to configure the cluster URL, username, and password (SHA256) provided by the Redpanda cluster.

    1. Below is a sample configuration snippet:

<?xml version="1.0" encoding="UTF-8" ?>
<kafka-configuration xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:noNamespaceSchemaLocation="config.xsd">
<internal>
<option>
<key>kafka.topic.default.replication-factor</key>
<value>3</value>
</option>
</internal>
<kafka-clusters>
<kafka-cluster>
<id>cluster01</id>
<bootstrap-servers>XXXXXX.prd.cloud.redpanda.com:9092</bootstrap-servers>
<authentication>
<scram-sha256>
<username>testuser</username>
<password>UwnXFy29qzVxOtKiEsTu7Cb761wkGl</password>
</scram-sha256>
</authentication>
<tls>
<enabled>true</enabled>
<hostname-verification>false</hostname-verification>
</tls>
</kafka-cluster>
</kafka-clusters>

</kafka-configuration>

  1. Enable the HiveMQ Kafka Extension:

    Remove the DISABLED file from the hivemq-kafka-extension folder to enable the extension.Start the hivemq broker

  2. Start the HiveMQ Broker:

  3. Confirmation of Extension Activation:

    Upon successful startup, verify hivemq logs that the extension is activated with the configured MQTT-to-Kafka mapping.
    Sample logs:

    2024-04-16 11:52:48,997 INFO - mKqRA: no members discovered after 2001 ms: creating cluster as first member 2024-04-16 11:52:49,323 INFO - Started mqtt-to-kafka mapping "mapping01" with MQTT topics [#] and Kafka topic "kafka-topic" for Kafka Cluster "cluster01" 2024-04-16 11:52:49,323 INFO - No mqtt-to-kafka transformers were found. No mqtt-to-kafka transformers are started. 2024-04-16 11:52:49,323 INFO - No kafka-to-mqtt mappings were found. No kafka kafka-to-mqtt mappings are started. 2024-04-16 11:52:49,324 INFO - No kafka-to-mqtt transformers were found. No kafka kafka-to-mqtt transformers are started. 2024-04-16 11:52:49,324 INFO - Extension "HiveMQ Enterprise Extension for Kafka" version 4.27.0 started successfully.

  4. Testing:

    Conduct a quick test by publishing messages and observing their transmission within the Redpanda cluster's console interface.

    1. image-20240416-095950.png

Following these steps ensures the successful integration of HiveMQ with a Redpanda cluster, enabling efficient data exchange and communication within the ecosystem.

Since we are using Redpanda free cloud cluster, we have added internal option configuration in our Kafka extension’s config.xml to set the replication factor as 3. This was required for a successful connection with Redpanda’s cloud serverless cluster.