Reducing Kafka Lag by Increasing Topic Partitions

In systems where HiveMQ Kafka Extension is used to publish MQTT messages to Kafka, high Kafka lag can result from insufficient consumer throughput. If the Kafka broker is accepting messages at the expected rate, but consumer processing is delayed, increasing the number of partitions for the affected topic is a practical solution.

 Instructions

1. Use Case

This procedure applies when:

• HiveMQ successfully publishes messages to Kafka topics.

• Kafka consumers are lagging behind in processing messages.

• Metrics indicate increasing lag despite stable broker performance.

2. Check Current Partition Count

To inspect the number of partitions for a given Kafka topic:

3. Increase Partitions

Kafka topics can only have their partition count increased, not decreased. Execute the following command to increase partitions:

4. Scale Consumer Applications

Kafka consumer clients process messages per partition. After increasing partitions:

• Ensure the consumer application is configured to handle additional threads or instances.

• For optimal parallelism, the number of active consumers in a group should match or exceed the number of partitions.

Failure to scale the consumer side appropriately will result in under-utilization of the new partitions.

6. Notes

• Partition count must be carefully planned to avoid excessive overhead in Kafka broker metadata and controller operations.

• Ensure the producer (HiveMQ Kafka Extension) uses a partitioning strategy that evenly distributes messages across partitions if key-based partitioning is used.

• Validate end-to-end latency to confirm that the increase in partitions yields the expected improvement in processing time.

References

• Apache Kafka Documentation: Apache Kafka

• HiveMQ Kafka Extension Guide: https://docs.hivemq.com/hivemq-kafka-extension/latest/index.html

 Related articles