/
Dropped Messages to Kafka Due to "Consumer Message Queue is Full" Error

Dropped Messages to Kafka Due to "Consumer Message Queue is Full" Error

Issue Summary

MQTT publish messages are dropped before being published to Kafka with the error message:

Outgoing publish message was dropped. Receiving consumer: kafka-extension_topic-transformer-1, topic: topic-name, qos: 1, reason: The consumer message queue is full.

This issue occurs intermittently (every few weeks) and requires a HiveMQ broker restart to temporarily restore normal data flow.

Symptoms

  • Messages drop with "consumer message queue is full" error

  • Queue grows from 1k to 60k+ messages, eventually reaching ~1 million messages

  • Issue recurs every few weeks after HiveMQ restart

  • Data flow resumes after restart but fails again after some time

Root Cause

The Kafka extension consumer cannot keep up with the message throughput from HiveMQ. The internal queue for the Kafka consumer grows faster than messages can be forwarded to Kafka, eventually reaching capacity and causing message drops.

This indicates backpressure where the incoming MQTT publish rate exceeds the forwarding rate to Kafka over time.

Identification Steps

1. Check HiveMQ Logs

Look for log entries containing:

  • "The consumer message queue is full"

  • "Outgoing publish message was dropped"

  • Consumer name (e.g., "kafka-extension_topic-transformer-1")

2. Monitor Queue Growth

Track the consumer message queue size over time to identify growth patterns.

3. Verify Kafka Cluster Health

Check for:

  • Performance degradation patterns

  • Resource constraints (CPU, memory, disk I/O)

  • Network connectivity issues

  • Kafka rebalancing events

4. Review Network Connectivity

Verify stable connectivity between HiveMQ and Kafka cluster, including:

  • Network latency

  • Throughput capacity

  • Any intermittent connectivity issues

Resolution Steps

Immediate Actions

  1. Monitor Kafka cluster health - Check for performance degradation patterns that align with the recurring cycle

  2. Review network connectivity - Verify stable connectivity between HiveMQ and Kafka cluster

  3. Implement alerting - Set up alerts when queued messages exceed thresholds (e.g., 10k, 30k, 50k)

Long-term Solutions

1. Upgrade HiveMQ Version

Upgrade to a supported HiveMQ version (4.52 or 4.28.26) as older versions may lack important improvements and features.

2. Investigate Kafka Performance

Review potential Kafka-side issues:

  • Kafka cluster performance degradation over time

  • Network connectivity issues between HiveMQ and Kafka

  • Kafka topic partition throttling or resource constraints

  • Kafka broker disk I/O bottlenecks

3. Review Kafka Extension Configuration

Verify and optimize:

  • Number of consumer threads configured

  • Batch size and linger time settings

  • Compression settings

  • Kafka producer buffer configurations