/
Troubleshooting High JVM Heap Memory Usage in HiveMQ

Troubleshooting High JVM Heap Memory Usage in HiveMQ

Troubleshooting High JVM Heap Memory Usage in HiveMQ

Issue Description

High JVM heap memory usage can lead to performance degradation and potential OutOfMemoryError crashes in HiveMQ. This issue may manifest as frequent memory spikes, particularly during high-load scenarios such as device reconnections, software updates, or increased message throughput.

Symptoms

  • Frequent Java heap memory usage spikes

  • OutOfMemory errors in logs

  • Broker crashes or restarts

  • Performance degradation during peak load times

  • Error messages such as "fatal error: OutOfMemory encountered: Java heap space"

Root Cause Analysis

High heap memory usage can be caused by:

  • Insufficient heap size allocation for workload requirements

  • Large number of concurrent client connections

  • High message throughput with queued messages

  • Retained messages accumulation

  • Extension memory consumption

Diagnostic Steps

Step 1: Monitor JVM Metrics

HiveMQ exposes JVM metrics through JMX that can help identify memory issues. Key metrics to monitor include:

  • com.hivemq.jvm.memory.heap.used - Current heap memory usage

  • com.hivemq.jvm.memory.heap.committed - Committed heap memory

  • com.hivemq.jvm.memory.heap.max - Maximum heap memory

  • com.hivemq.jvm.gc.collection-time - Garbage collection time

  • com.hivemq.jvm.gc.collection-count - Garbage collection count

Step 2: Check Message Queue Metrics

Monitor message-related metrics that can contribute to memory pressure:

  • com.hivemq.messages.client.queued.count - Current number of queued messages

  • com.hivemq.messages.client.pending.qos-0.count - Pending QoS 0 messages

  • com.hivemq.messages.client.pending.total.count - Total pending messages

  • com.hivemq.messages.dropped.qos-0-memory-exceeded.count - Messages dropped due to memory limits

Step 3: Review Extension Metrics

Check extension-related memory consumption:

  • com.hivemq.extension.managed-executor.running - Running extension jobs

  • com.hivemq.extension.managed-executor.submitted - Rate of submitted extension jobs

A full list of available metrics and monitoring solutions can be found on our monitoring documentation page here.

 

Resolution Steps

Option 1: Increase Heap Size

If monitoring indicates consistent high memory usage, increase the JVM heap size allocation. The default heap size may be insufficient for your workload.

Option 2: Implement Monitoring with Prometheus

Install the HiveMQ Prometheus Monitoring Extension to track memory metrics over time:

  1. Download the HiveMQ Prometheus Monitoring Extension

  2. Unpack and move to the extensions folder

  3. Configure prometheusConfiguration.properties

  4. Access metrics at <ip>:<port><metric_path>

Option 3: Implement Monitoring with InfluxDB

Install the HiveMQ InfluxDB Monitoring Extension for time-series monitoring:

  1. Download the HiveMQ InfluxDB Monitoring Extension

  2. Move to the extensions folder

  3. Configure influxdb.properties with required host and port properties

Prevention Strategies

1. Establish Baseline Monitoring

  • Set up continuous monitoring using Prometheus or InfluxDB extensions

  • Track heap memory usage patterns over time

  • Establish alerts for memory thresholds

2. Capacity Planning

  • Monitor metrics during peak load scenarios

  • Size heap memory appropriately for your workload

3. Load Testing

  • Test memory behavior during high-load scenarios such as mass reconnections

  • Validate heap sizing before production deployments

  • Test after version upgrades to identify memory requirement changes, if upgrading from an older version. Note that any changes of this nature are well-documented within version upgrade details, and can be verified first before upgrade.

Additional Resources