Task Queue Metrics to identify the cause for Overload Protection

HiveMQ Overload protection uses the total number of tasks to ascertain when to become active and throttle incoming work.

While growing task queues are generally a sign of overload on a broker, short spikes are normal and not an immediate reason to worry.

HiveMQ uses task queues to manage and execute asynchronous tasks efficiently. It receives incoming work, splits it into tasks and adds them to a queue which is processed one by one in a controlled and orderly manner. Each task is typically a discrete unit of work and can be executed independently.

The tasks can be grouped into the following 4 categories:

Persistence tasks

These tasks relate to disk reads and writes.

Total tasks

com.hivemq.persistence-executor.queued com.hivemq.persistence.executor.total.tasks

Connection establishment tasks

com.hivemq.persistence_executor.client-events.tasks com.hivemq.persistence_executor.client-session.tasks

Messaging tasks

com.hivemq.persistence_executor.incoming-message-flow.tasks com.hivemq.persistence_executor.queued.messages.tasks

Single-writer-callback tasks

These tasks relate to data replication between individual cluster members

com.hivemq.single-writer-callback-executor.*.queued

Netty tasks

These tasks are related to Network Input-Output throughput

Extension tasks

These tasks denote SDK calls made by extensions

com.hivemq.extension.task.executors.queued

How to query these metrics?

com_hivemq_extension_task_executors_queued

com_hivemq_persistence_executor_queued
{__name__=~"single-writer-callback-executor_(.*)_queued"}
com_hivemq_persistence_executor_total_tasks

com_hivemq_persistence_executor_client_events_tasks

com_hivemq_persistence_executor_client_session_tasks

com_hivemq_persistence_executor_queued_messages_tasks

com_hivemq_persistence_executor_incoming_message_flow_tasks

com.hivemq.extension.task.executors.queued

com.hivemq.persistence-executor.queued

/com.hivemq.persistence.executor.*.tasks/