Setup Kafka and HiveMQ Kafka Extension for Windows Standalone Server 2022

Prerequisite

• Running windows server E.g. EC2 installation https://hivemq.atlassian.net/wiki/spaces/KB/pages/2728263766

• HiveMQ is installed https://hivemq.atlassian.net/wiki/spaces/KB/pages/2743140425

(Optional) Updating the Security group TCP ports of EC2 instance

• If you are operating Windows Server on EC2 and require external access to its services, it is necessary to add the following ports to the related security group within the Inbound rules. This step ensures that the services hosted on your server can be accessed from outside the EC2 environment.

  • 9092 > Kafka default port

  • 2181 > Zookeeper default port

  • 3389 > Remote Desktop Protocol

  • 1883,8883 > MQTT Ports

• Please also check your firewall rules to make sure these ports are not blocked for local communication.

Install Kafka on Windows Server 2022

Step 1: Connect to Your Windows Machine - EC2 Instance (Optional)

1. Login to the windows machine. Connect to your EC2 instance using Remote Desktop Protocol (RDP). You'll need your instance's public IP address and the password for the Administrator account.

Step 2: Install Java

Kafka requires Java to run, so the first step is to install a Java Runtime Environment (JRE) or Java Development Kit (JDK).

1. Download Java: Download the Java JDK installer for Windows.

2. Install Java:

   1. Run the downloaded installer and follow the instructions to install Java on your server.

   2. (Recommended) Set the Environmental variable during installation can be done by enabling the following setting

      Open

      

   3. Set JAVA_HOME Environment Variable manually:

      • Right-click on "This PC" and select "Properties".

      • Click on "Advanced system settings".

      • In the System Properties window, go to the "Advanced" tab and click on "Environment Variables".

      • Under "System Variables", click "New" and add a new variable:

        • Variable name: JAVA_HOME

        • Variable value: the path to your Java installation (e.g., C:\Program Files\Eclipse Adoptium\jdk-11.0.22.7-hotspot\).

        • C:\Program Files\Eclipse Adoptium\jdk-11.0.22.7-hotspot\bin exists under path system variable list.

      • Click OK to close all dialogues.

Step 3: Download and Install Kafka

1. Download Kafka: Visit the Apache Kafka website to download the latest version of Kafka for Windows. Choose the binary files suitable for Windows.

2. Extract Kafka: Once downloaded, extract the Kafka archive to a location on your server, such as C:\kafka. You can use the following command via CMD to unzip the file e.g. tar -xvzf hivemq-4.26.0.tgz

3. In case you extracted the file in some other folder, please rename and move the folder to C:\kafka

Step 4: Configure Kafka

1. Configure Server Properties:

   • Navigate to your Kafka directory (e.g., C:\kafka) and then to the config folder.

   • Edit the server.properties file using a text editor like Notepad.

   • Find the line that specifies listeners=PLAINTEXT://:9092 and ensure it is uncommented (remove the # at the beginning if present).

   • Optionally, adjust the logs line to fileslog.dirs=C:\kafka\kafka-logs

2. Configure the Zookeeper properties

   • Navigate to your Kafka directory (e.g., C:\kafka) and then to the config folder.

   • Edit the zookeeper.properties file using a text editor like Notepad.

   • adjust the dataDir line to DataDir=C:\kafka\zookeeper-data

Step 5: Start Kafka

1. Start the Zookeeper Service:

   • Open a command prompt as Administrator.

   • Navigate to your Kafka directory.

   • Run the command: .\bin\windows\zookeeper-server-start.bat .\config\zookeeper.properties.

   • Keep this command prompt open.

     Open

     

2. Start the Kafka Server:

   • Open a new command prompt as Administrator.

   • Navigate to your Kafka directory.

   • Run the command: .\bin\windows\kafka-server-start.bat .\config\server.properties.

   • Kafka should now be running on your server.

     Open

     

Step 6: Verify the Installation

1. Create a Topic:

   • Open another command prompt as Administrator.

   • Navigate to your Kafka directory.

   • Create a new topic with the command e.g. test in this case: .\bin\windows\kafka-topics.bat --create --topic test --partitions 1 --replication-factor 1 --bootstrap-server
localhost:9092

   • In case you are using the older version of Kafka, the following command does the same action .\bin\windows\kafka-topics.bat --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test.

2. Test Producing and Consuming Messages:

   • Produce a message: In the same command prompt, enter the command:

     .\bin\windows\kafka-console-producer.bat --broker-list localhost:9092 --topic test

     and then type a message e.g. your name.

     Open

     

   • Consume a message: Open another command prompt, navigate to the Kafka directory, and enter the command: .\bin\windows\kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic test --from-beginning to read the message.

     Open

     

Configure HiveMQ to Use Kafka

1. Go to your HiveMQ Kafka extension folder C:\hivemq\extensions\hivemq-kafka-extension and delete the DISABLED file

2. Create the following sample file in C:\hivemq\extensions\hivemq-kafka-extension\conf called config.xml. (Please note that we set two different Kafka topics, so the setup does not run into a loop.)

   <?xml version="1.0" encoding="UTF-8" ?> <kafka-configuration xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="config.xsd"> <kafka-clusters> <kafka-cluster> <id>cluster01</id> <bootstrap-servers>127.0.0.1:9092</bootstrap-servers> </kafka-cluster> </kafka-clusters> <mqtt-to-kafka-mappings> <mqtt-to-kafka-mapping> <id>mapping01</id> <cluster-id>cluster01</cluster-id> <mqtt-topic-filters> <mqtt-topic-filter>#</mqtt-topic-filter> </mqtt-topic-filters> <kafka-topic>test</kafka-topic> </mqtt-to-kafka-mapping> </mqtt-to-kafka-mappings> <kafka-to-mqtt-mappings> <kafka-to-mqtt-mapping> <id>mapping02</id> <cluster-id>cluster01</cluster-id> <kafka-topics> <kafka-topic>test2</kafka-topic> </kafka-topics> </kafka-to-mqtt-mapping> </kafka-to-mqtt-mappings> </kafka-configuration>

3. Start hivemq run.bat file as an administrator under C:\hivemq\bin and verify the Kafka startup logs

   Open

   

Test MQTT To Kafka Mapping

1. Open the CMD terminal on C:\hivemq\tools\mqtt-cli\bin and publish the following
   mqtt.bat pub --topic test --message "Hello" --host localhost --port 1883

2. Open a command prompt, navigate to the Kafka directory C:\kafka, and enter the command: .\bin\windows\kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic test --from-beginning to read the message.

3. Check the consumer, it should show the sent message

   Open

   

Test Kafka To MQTT Mapping

1. Under C:\kafka, If it does not already exist, Create a Kafka topic “test2” via the following command (this is the value of kafka topic in config.xml under kafka extension conf folder)

   .\bin\windows\kafka-topics.bat --create --topic test2 --partitions 1 --replication-factor 1 --bootstrap-server localhost:9092

2. Open the CMD terminal on C:\hivemq\tools\mqtt-cli\bin and subscribe to the following

   mqtt.bat sub --topic # --host localhost --port 1883 -J

3. Under C:\kafka , create a message via Kafka publisher e.g. Hello

   .\bin\windows\kafka-console-producer.bat --broker-list localhost:9092 --topic test2

    

   Open

   

4. Check the MQTT Subscriber, the message should be received

Open