Get information about file descriptors used by system
This article helps you to find out information about file descriptors used by the system.
Prerequisite:
An exception as “Too many files open”
is observed in your hivemq.log
and max open file limit is already set as “1000000
“ files.
Instructions
Step 1:
There are two ways to get a current an open and max file descriptor(FD) count.
Enable monitoring with HiveMQ Prometheus Extension. The highly-performant metrics subsystem of HiveMQ lets you monitor relevant metrics with no reduction in system performance (even in low-latency high-throughput environments).
In the context of this article you can check following metrics:com.hivemq.system.open-file-descriptor
com.hivemq.system.max-file-descriptor
These metrics provide counts of currently open file descriptors and max file descriptors.Also, as an alternative, you can use the below commands from your shell to get the requested metrics
The following command displays the soft limit, hard limit, and units of measurement for each of the process's resource limits. From that list, you can get details about
Max open files
$ cat /proc/${PID}/limits
where${PID}
is your hivemq broker's process id.The following command can be used to know how many file descriptors are being used
$ cat /proc/sys/fs/file-nr
You can interpret the file content as
column 1 = total allocated file descriptors (the number of file descriptors allocated since boot)
column 2 = total free allocated file descriptors
column 3 = maximum open file descriptors$ ulimit
to know the number of open file descriptors per process
Step 2
Please execute the following steps to get the list of open files.
Install
lsof
if it’s not available already else skip this step$ apt update && apt install lsof
switch to hivemq user (UID of the process)
$ su hivemq
show all open files
$ lsof -p ${PID} > open_files_pid_${PID}.txt
Share created open_files.txt
with HiveMQ support to investigate further.
How to increase Open File Descriptors Limits in Ubuntu
User File Descriptors Limits
Per-user are set in the following file: /etc/security/limits.conf
The file has the following syntax:
<domain> <type> <item> <value>
Setting soft and hard file limits for user hivemq:
hivemq soft file 1000000
hivemq hard file 1000000
Checking User File Limits
User soft file limit:
ulimit -Su
User hard file limit:
Checking for another user (hivemq
):
Kernel File Descriptors Limits
System wide open file descriptor limits are set with sysctl. For example, increase the limit to 1000000 open file descriptors:
Note, this will only work until the next reboot.
Check your work:
To make permanent changes (persisted after the reboot), it is necessary to edit the /etc/sysctl.conf
Add the following line:
Users will need to logout and login again for the changes to take effect. If you want to apply the limit immediately, you can use the following command:
Related articles
https://www.tecmint.com/set-limits-on-user-processes-using-ulimit-in-linux/