Client Event History performance decreases over time

Owned by Daria Samkova
May 02, 2023
1 min read

Expected behavior

The Client Event History feature adds fine-grained, per-client observability to the HiveMQ Control Center and provides a historical view of client events. The event history makes it easier to understand what happened with a single client and helps you diagnose problems, even at a later time.

Observed behavior

A HiveMQ cluster node, after a short network disruption, is rejoining the cluster, however, the merge is taking a long time and the metric com_hivemq_persistence_executor_client_events_tasks is increasing:

Reason

When brokers are running for a long time without a restart, and the Client Event History is active, the persistence of the broker contains too many so-called ā€œtombstonesā€, which are used to maintain data consistency in case of a cluster split.

This might lead to longer timers for the broker to handle persistence client event history tasks, for example, during a cluster merge.

Affected Version

<=4.9.3, 4.12

Solution / Workaround

  • Disabling the <client-event-history> in the /opt/hivemq/conf/config.xml. If this is not possible, then:

  • Regularly perform a rolling restart of HiveMQ cluster nodes, which will clean up ā€œtombstonesā€ and improve client-event-history tasks times.