Confluent Kafka connector producing messages only on partition 0

Owned by Harsh Bansal
Sept 06, 2023
1 min read

Question

I have integrated HiveMQ as a source connector to my confluent Kafka queue, but all the messages are being produced only on partition 0, how can I make HiveMQ produce to all partitions (I have 6)?

Answer

In the context of Kafka integrations, the default behaviour designates the MQTT topic as the partitioning key. As a result, when messages are dispatched to the same topic, they are also directed to the same partition. This aligns with the anticipated functionality of the Kafka extension, ensuring the sequential order of messages.

While your current setup receives all messages on a single partition due to having just one MQTT topic configured, it's worth noting that the capability to modify partitions is achievable through customizations.

However, it's important to mention that this feature is not currently available within the Free/Starter Cloud offering. In case you are interested in upgrading to Cloud Enterprise please let us know, our sales team will be interested in connecting with you.