Scrape prometheus metrics on EC2 instance using HiveMQ Prometheus Extension

Prerequisite

1. A running EC2 instance.

2. A running hivemq setup

3. ssh and sudo access to the ec2 instance

Prometheus Setup

Install Prometheus on EC2 instance

1. SSH into your ec2 instance and switch to root user

2. Download Prometheus (e.g. wget …) and untar the file(tar -xzfv …)

3. Rename the unzipped folder. e.g.
   mv prometheus-2.22.0.linux-amd64 prometheus-files

4. Create a Prometheus user, required directories, and make Prometheus the user as the owner of those directories.

   sudo useradd --no-create-home --shell /bin/false prometheus
   sudo mkdir /etc/prometheus
   sudo mkdir /var/lib/prometheus
   sudo chown prometheus:prometheus /etc/prometheus
   sudo chown prometheus:prometheus /var/lib/prometheus

5. Copy prometheus and promtool binary from prometheus-files folder to /usr/local/bin and change the ownership to prometheus user.

sudo cp prometheus-files/prometheus /usr/local/bin/
sudo cp prometheus-files/promtool /usr/local/bin/
sudo chown prometheus:prometheus /usr/local/bin/prometheus
sudo chown prometheus:prometheus /usr/local/bin/promtool

6. Move the consoles and console_libraries directories from prometheus-files to /etc/prometheus folder and change the ownership to prometheus user.

sudo cp -r prometheus-files/consoles /etc/prometheus
sudo cp -r prometheus-files/console_libraries /etc/prometheus
sudo chown -R prometheus:prometheus /etc/prometheus/consoles
sudo chown -R prometheus:prometheus /etc/prometheus/console_libraries

Setup Prometheus Configuration

All the prometheus configurations should be present in /etc/prometheus/prometheus.yml file.

1. Create the prometheus.yml file.

sudo vi /etc/prometheus/prometheus.yml

2. Copy the following contents to the prometheus.yml file.

global:
  scrape_interval: 10s

scrape_configs:
  - job_name: 'prometheus'
    scrape_interval: 5s
    static_configs:
      - targets: ['localhost:9090']

3. Change the ownership of the file to prometheus user.

sudo chown prometheus:prometheus /etc/prometheus/prometheus.yml

Setup Prometheus Service File

Step 1: Create a prometheus service file.

sudo vi /etc/systemd/system/prometheus.service

Step 2: Copy the following content to the file.

[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
    --config.file /etc/prometheus/prometheus.yml \
    --storage.tsdb.path /var/lib/prometheus/ \
    --web.console.templates=/etc/prometheus/consoles \
    --web.console.libraries=/etc/prometheus/console_libraries

[Install]
WantedBy=multi-user.target

Step 3: Reload the systemd service to register the prometheus service and start the prometheus service.

sudo systemctl daemon-reload
sudo systemctl start prometheus

Check the prometheus service status using the following command.

sudo systemctl status prometheus

The status should show the active state.

Access Prometheus Web UI

Now you will be able to access the prometheus UI on 9090 port of the prometheus server.

http://<ec2-auto-assigned-ip>:9090/graph

In case of EC2, replace this IP with Auto-assigned IP address

Setup HiveMQ Prometheus Operator

1. On your ec2 instance, go to your hivemq folder and cd into the extensions folder

2. Download and unzip the latest HiveMQ Prometheus extension. The default configuration is already set in prometheusConfiguration.properties. This usually does not need to be changed

3. Start hivemq by executing run.sh under hivemq/bin folder(please make sure its executable(chmod +x run.sh)

4. You can confirm if the extension worked well by

   1. checking hivemq logs

      

   2. Go to http://<ec2-auto-assigned-ip>/metrics, it should show scrapped metrics page
      

      

Connecting Prometheus to scrape HiveMQ Metrics

1. Access Your Prometheus Configuration File: Log into your EC2 instance where Prometheus is running. You'll typically find the prometheus.yml configuration file in the directory where you have Prometheus installed, or under /etc/prometheus/ if you used a package manager to install Prometheus.

2. Edit the Configuration File: Open prometheus.yml in a text editor. You can use nano, vi, or any editor you're comfortable with. For example:

   sudo vi /etc/prometheus/prometheus.yml

3. Add a New Job to Scrape HiveMQ Metrics: In the scrape_configs section of the configuration file, add a new job to scrape metrics from the HiveMQ Prometheus operator. You'll specify the metrics path and the address of the server where HiveMQ is running. For example we used ec2 IP and port 9399 for HiveMQ operator:

   scrape_configs:
     - job_name: 'hivemq'
       static_configs:
         - targets: ['<ec2-auto-assigned-ip>:9399']
       metrics_path: '/metrics'

4. Save and Close the File: After adding the new job, save your changes and exit the text editor.

5. Reload Prometheus Configuration: Prometheus needs to reload its configuration to start scraping metrics from the new endpoint. You can do this by sending a SIGHUP signal to the Prometheus process. If you're using systemd, you can reload the Prometheus service using:

   sudo systemctl reload prometheus

   If you're not using systemd, find the Prometheus process ID and send a SIGHUP signal to it:

   kill -HUP $(pidof prometheus)

6. Verify Configuration: After reloading the configuration, you can check that Prometheus is successfully scraping metrics from HiveMQ by accessing the Prometheus UI and navigating to the ‘Targets’ page.(http://<ec2-auto-assigned-ip>/targets) You should see your new job listed with a status indicating whether the scrape is successful.

   

7. Going back to http://<ec2-auto-assigned-ip>/graph, you should be able to see HiveMQ metrics scrapped.

   

This concludes the tutorial. As next steps you can connect prometheus to Grafana for better visualisation and dashboards