/
Decreased Message Throughput During MQTT Broker Deployment

Decreased Message Throughput During MQTT Broker Deployment

Jan 28, 2026

Problem

During deployment on the MQTT Broker, a decrease in incoming and outgoing message throughput was observed.

Symptoms

  • Reduced message throughput (incoming/outgoing messages per second)

  • Performance degradation during deployment operations

  • Potential impact on connected clients and message delivery

Diagnostic Steps

To properly diagnose throughput issues during deployment, collect the following information:

1. Collect HiveMQ Metrics

HiveMQ's metrics are essential for analyzing deployment behaviour and throughput patterns. Export metrics from Prometheus or Grafana to analyze the full set of performance indicators.(1)(https://hivemq.atlassian.net/wiki/spaces/KB/pages/2711486465/HiveMQ+Support+-+Customer+Onboarding+Readiness+Process+Tabular+Format)

2. Review HiveMQ Logs

Examine the following log files on all cluster nodes:(1)(https://hivemq.atlassian.net/wiki/spaces/KB/pages/2711486465/HiveMQ+Support+-+Customer+Onboarding+Readiness+Process+Tabular+Format)

  • $HIVEMQ_HOME/log/hivemq.log

  • $HIVEMQ_HOME/log/event.log

  • $HIVEMQ_HOME/log/migration.log

These logs provide information about:

  • Overload Protection activation

  • Cluster member joins/departures

  • Rolling upgrade status

  • Performance-related events

3. Create Thread Dump Series

If resource congestion cannot be explained through logs and metrics, capture thread dump series to identify heavy consumers and processing bottlenecks.(1)(https://hivemq.atlassian.net/wiki/spaces/KB/pages/2711486465/HiveMQ+Support+-+Customer+Onboarding+Readiness+Process+Tabular+Format)

4. Generate Diagnostic Archive

For HiveMQ version 4.2x or greater, generate a Diagnostic Archive from the HiveMQ Control Center. This includes:(1)(https://hivemq.atlassian.net/wiki/spaces/KB/pages/2711486465/HiveMQ+Support+-+Customer+Onboarding+Readiness+Process+Tabular+Format)

  • Cluster member information

  • Configuration details (passwords redacted)

  • Thread dump series

  • Short snippet of HiveMQ metrics

Common Causes and Solutions

Cluster Communication During Deployment

During deployment operations, intra-cluster communication patterns may change, potentially affecting throughput. The cluster transport handles replication, metrics collection, and maintenance functions.(2)(https://www.hivemq.com/blog/hivemq-4-18-delivers-higher-mqtt-throughput)

Resource Constraints

Verify that system requirements are met:(1)(https://hivemq.atlassian.net/wiki/spaces/KB/pages/2711486465/HiveMQ+Support+-+Customer+Onboarding+Readiness+Process+Tabular+Format)

  • Minimum CPU/RAM criteria

  • At least 50% of RAM available

  • Adequate heap configuration

Overload Protection Activation

Check logs for Overload Protection activation, which may throttle connections or apply TCP back-pressure during high load.(1)(https://hivemq.atlassian.net/wiki/spaces/KB/pages/2711486465/HiveMQ+Support+-+Customer+Onboarding+Readiness+Process+Tabular+Format)

Resolution Steps

  1. Verify Rolling Upgrade Process - Ensure rolling upgrades are performed correctly, waiting for each node to be ready before proceeding to the next node (as indicated in logs)(1)(https://hivemq.atlassian.net/wiki/spaces/KB/pages/2711486465/HiveMQ+Support+-+Customer+Onboarding+Readiness+Process+Tabular+Format)

  2. Monitor Cluster Stability - Check that all cluster nodes have rejoined and are functioning properly after deployment

  3. Review Resource Utilization - Confirm adequate CPU, memory, and network capacity during deployment operations

  4. Check Downstream Systems - Verify that databases and analytical applications can handle the message load from the broker

  5. Analyze Metrics - Review exported metrics to identify specific bottlenecks or resource constraints

Prevention and Best Practices

Maintain steady resource utilization during your upgrade

Before you shut down a node with an old version, we recommend that you add a node to your cluster that is configured with the target version. This method is particularly important for clusters that have high resource utilization. For example, if your cluster runs with over 60% CPU usage. When you remove a node from your cluster during the upgrade process, the workload of the node you remove is redistributed over the nodes that remain in the cluster. This redistributed workload increases the amount of work each remaining node must do. The addition of a new node, before you remove any of the existing nodes, helps you maintain a steady workload in your cluster as you upgrade each node to the new version. Once all previously-existing nodes are upgraded, you can remove the node that you added to the cluster.

Deployment Planning

  • Schedule deployments during low-traffic periods when possible

  • Ensure adequate capacity headroom before deployment

  • Test deployment procedures in non-production environments first

Monitoring

Capacity Management

Additional Resources

For further assistance with throughput issues:

Related Information

—

Labels: kb-troubleshooting-article, throughput, deployment, performance, mqtt-broker
```