/
Disk Usage Increasing Without Being Released in HiveMQ 4.44-4.46

Disk Usage Increasing Without Being Released in HiveMQ 4.44-4.46

Overview

This article documents a known critical issue affecting HiveMQ versions 4.44 through 4.46 where disk usage continuously increases without being properly released, particularly in the publish_payload_store directory. This issue is especially prevalent in Kubernetes deployments with specific CPU configurations.

Symptoms

  • Continuous disk usage growth in /opt/hivemq/data/persistence/publish_payload_store

  • Disk usage increases even with minimal queued messages and retained messages

  • Publish payloads are not cleaned up correctly

  • Issue persists despite normal broker operations

Root Cause

HiveMQ versions 4.44 to 4.46 contain a critical configuration issue that can cause excessive disk space usage on nodes with non-power-of-two CPU core counts(1)(https://docs.hivemq.com/hivemq/latest/upgrade/4-44-to-4-45.html). The issue involves persistence cleanup where publish payloads are not cleaned up correctly on deployments with certain CPU configurations(1)(https://docs.hivemq.com/hivemq/latest/upgrade/4-44-to-4-45.html).

In Kubernetes environments, the effective CPU quota visible inside the container through cgroups determines the behavior, not just the underlying instance type. For example, a pod CPU limit of 2500m (2.5 cores) can trigger this issue even on nodes with 4 vCPUs available.

Affected Versions

  • HiveMQ 4.44.0 through 4.46.0

  • All deployment types (self-managed, Kubernetes, bare metal)

Resolution

Permanent Fix

Upgrade to HiveMQ 4.46.1 or higher, which resolves this issue(1)(https://docs.hivemq.com/hivemq/latest/upgrade/4-44-to-4-45.html).

For upgrade procedures, follow the rolling upgrade process documented for HiveMQ clusters(2)(https://docs.hivemq.com/hivemq/latest/user-guide/cluster.html).

Immediate Mitigation (Before Upgrade)

If immediate upgrade is not possible, implement the following workaround:

1. Increase CPU Limits

For Kubernetes deployments using Helm, increase the HiveMQ pod CPU limit to at least 4 cores(3)(https://docs.hivemq.com/hivemq-platform-operator/configuration.html):

resources: cpu: 4000m

2. Perform Rolling Restart

Execute a rolling restart of cluster nodes one by one to allow cleanup processes to run with increased resources(2)(https://docs.hivemq.com/hivemq/latest/user-guide/cluster.html).

Post-Upgrade Actions

After upgrading to 4.46.1 or higher:

  1. Monitor disk usage - note that physical disk space reclamation occurs during flush/compaction cycles and may not be immediate

  2. Check the cluster-backup subfolder - if it is large and the cluster is healthy, it can be safely removed

  3. Verify that disk usage stabilizes at expected levels

Important Warnings

DO NOT manually delete publish_payload_store or other persistence folders on a running node. This can lead to:

  • Data inconsistencies

  • Missing payload references

  • Cluster instability

Prevention

To prevent recurrence:

  1. Upgrade Policy: Maintain HiveMQ at version 4.46.1 or higher

  2. CPU Allocation: In Kubernetes environments, allocate CPU limits as power-of-two values (2, 4, 8 cores) when possible

  3. Monitoring: Implement disk usage monitoring with alerts for the persistence directories(4)(https://docs.hivemq.com/hivemq/latest/user-guide/monitoring.html)

  4. Regular Maintenance: Follow HiveMQ maintenance best practices and stay current with maintenance releases

Additional Resources

Related Issues

This issue may appear alongside other known issues in HiveMQ 4.44-4.46:

For assistance, contact HiveMQ Support.

Last Updated: Based on HiveMQ 4.46.1 release information

Applies To: HiveMQ Self-Managed versions 4.44.0 - 4.46.0