How to set the Kafka consumer max.partition.fetch.bytes

Owned by Daria Samkova, created
Jun 12, 2024
2 min read

HiveMQ, with its Kafka extension, acts as a Kafka consumer. By default, Kafka consumers have a message size limit of 1MB. This is controlled by the max.partition.fetch.bytes configuration parameter in Apache Kafka consumers. Adjusting this setting can be crucial for applications that require consuming larger messages from Kafka topics.

This article will guide you through the process of modifying the max.partition.fetch.bytes setting in the HiveMQ Kafka Extension configuration file.

Prerequisites

  • HiveMQ with the Kafka Extension installed.

  • Access to the HiveMQ server filesystem, specifically to the Kafka Extension's configuration directory.

 Instructions

  1. Locate the Configuration File. The Kafka Extension configuration file is typically located at: /opt/hivemq/extensions/hivemq-kafka-extension/conf/config.xml

  2. Edit the Configuration File. To adjust the max.partition.fetch.bytes setting, you need to edit the config.xml file. You can use any text editor of your choice

  3. Locate the <kafka-configuration> section in the file. It may look something like this:

    <kafka-configuration> ... ... </kafka-configuration>

  4. Add the following lines within the <internal> section of the <kafka-configuration> block:

    <internal> <option> <key>kafka.consumer.consumer.poll.size.partition</key> <value>xxx</value> </option> </internal>

    Replace xxx with the desired value for max.partition.fetch.bytes in bytes. For example, to set it to 2MB, you would use 2097152.

  5. Example Configuration. Here’s an example of how the configuration might look after adding the new setting:

    <kafka-configuration> ... <internal> <option> <key>kafka.consumer.consumer.poll.size.partition</key> <value>2097152</value> <!-- 2MB --> </option> </internal> ... </kafka-configuration>

  1. Save and Close the File. After editing the file, save your changes and close the text editor.

  2. Verification. Monitor the Kafka consumer behavior to ensure it is now able to handle larger messages as per the new max.partition.fetch.bytes setting

By following these steps, you can successfully adjust the max.partition.fetch.bytes parameter for the Kafka consumer in the HiveMQ Kafka Extension. This allows you to handle larger messages from Kafka topics, ensuring your HiveMQ setup can cater to your specific needs.

 Related articles