Get information about file descriptors used by system

Owned by Sheetal Shet, created
Last updated: Feb 20, 2023 by Daria Samkova
2 min read

This article helps you to find out information about file descriptors used by the system.

Prerequisite:

An exception as “Too many files open” is observed in your hivemq.log
and max open file limit is already set as “1000000“ files.

Instructions

Step 1:

There are two ways to get a current an open and max file descriptor(FD) count.

  1. Enable monitoring with HiveMQ Prometheus Extension. The highly-performant metrics subsystem of HiveMQ lets you monitor relevant metrics with no reduction in system performance (even in low-latency high-throughput environments).

    In the context of this article you can check following metrics:
    com.hivemq.system.open-file-descriptor
    com.hivemq.system.max-file-descriptor

    These metrics provide counts of currently open file descriptors and max file descriptors.

  2. Also, as an alternative, you can use the below commands from your shell to get the requested metrics

    1. The following command displays the soft limit, hard limit, and units of measurement for each of the process's resource limits. From that list, you can get details about Max open files
      $ cat /proc/${PID}/limits
      where ${PID} is your hivemq broker's process id.

    2. The following command can be used to know how many file descriptors are being used
      $ cat /proc/sys/fs/file-nr
      You can interpret the file content as
      column 1 = total allocated file descriptors (the number of file descriptors allocated since boot)
      column 2 = total free allocated file descriptors
      column 3 = maximum open file descriptors

    3. $ ulimit to know the number of open file descriptors per process

Step 2

Please execute the following steps to get the list of open files.

  1. Install lsof if it’s not available already else skip this step

    • $ apt update && apt install lsof

  2. switch to hivemq user (UID of the process)

    • $ su hivemq

  3. show all open files

    • $ lsof -p ${PID} > open_files_pid_${PID}.txt

Share created open_files.txt with HiveMQ support to investigate further.

How to increase Open File Descriptors Limits in Ubuntu

User File Descriptors Limits

Per-user are set in the following file: /etc/security/limits.conf

The file has the following syntax:

<domain> <type> <item> <value>

Setting soft and hard file limits for user hivemq:

hivemq soft file 1000000 hivemq hard file 1000000

Checking User File Limits

User soft file limit:

ulimit -Su

User hard file limit:

Checking for another user (hivemq):

Kernel File Descriptors Limits

System wide open file descriptor limits are set with sysctl. For example, increase the limit to 1000000 open file descriptors:

Note, this will only work until the next reboot.

Check your work:

To make permanent changes (persisted after the reboot), it is necessary to edit the /etc/sysctl.conf

Add the following line:

Users will need to logout and login again for the changes to take effect. If you want to apply the limit immediately, you can use the following command:

https://www.tecmint.com/increase-set-open-file-limits-in-linux/#How_to_Check_System_wide_File_Descriptors_Limits_in_Linux

https://www.tecmint.com/set-limits-on-user-processes-using-ulimit-in-linux/