Troubleshooting Datadog HiveMQ metrics | HiveMQ Platform Kubernetes Operator (new)

Problem

I followed Expose HiveMQ Prometheus metrics in DataDog using the HiveMQ operator but not seeing any metrics in the Datadog HiveMQ Dashboard.

Solution

Verify Annotations

Check that the annotations that you added for Datadog to the hivemq-platform pods' annotations have the correct openmetrics_endpoint pattern

nodes:
  annotations:
    ad.datadoghq.com/hivemq.check_names: '["openmetrics"]'
    ad.datadoghq.com/hivemq.init_configs: '[{}]'
    ad.datadoghq.com/hivemq.instances: |
      [
        {
          "openmetrics_endpoint": "http://%%host%%:9399/",
          "max_returned_metrics": 10000,
          "metrics": [".*"]
        }
      ]

Verify that Datadog Agent is running and healthy

Check the logs of the Datadog Agent and verify there are no errors.

kubectl logs -n datadog datadog-agent-abcd1

Verify that Datadog Agent is reaching the openmetrics_endpoints

Run the hivemq-platform with nodes.logLevel=DEBUG. In the DEBUG log level, the hivemq.log will reflect the GET requests from the Datadog Agent.
1. How to run hivemq-platform with nodes.logLevel=DEBUG?
  1. Helm upgrade:
    helm upgrade <your-hivemq-platform-release-name> hivemq/hivemq-platform --values <your-values.yaml> --set nodes.logLevel=DEBUG --namespace hivemq
  2. Or hivemq-platform’s your-values.yaml:
    nodes: logLevel: DEBUG
After hivemq-platform starts, monitor the DEBUG log messages about GET requests to port 9399. Example log message:
DEBUG - Received HTTP-Get-Request from Prometheus to scrape metrics from HiveMQ: Request(GET http://10.244.0.66:9399/)@1602027c
If you are not seeing GET requests to port 9399 in the log, it indicates the Datadog Agent is not scraping any metrics from the hivemq-platform pods.

Verify the Datadog Dashboard

There are two methods for the Datadog to ingest HiveMQ metrics: one is for the JMS metrics, one for the Open Telemetry (Prometheus) metrics. JMS metrics are named like com.hivemq.messages.count while Open Telemetry (Prometheus) metrics are named like com_hivemq_messages_count. Verify that the Dashboard refers to the metrics named like com_hivemq_messages_count.