Systemd >=240 version defaults Max open files to 65535

Observation

The hivemq.log shows statements like the following:

Error reading file /proc/stat. {}java.nio.file.FileSystemException: /proc/stat: Too many open files at

Consequence

The impact of this is severe.

From Wikipedia (https://en.wikipedia.org/wiki/File_descriptor )

In Unix and related computer operating systems, a file descriptor (FD, less frequently fildes) is an abstract indicator (handle) used to access a file or other input/output resource, such as a pipe or network socket.

The most immediate impact this will have for HiveMQ is that the broker cannot accept any additional connection, as there are not file descriptors for network sockets left.
This might also impact the ability to write log files or replicate data.

Cause

This is caused when HiveMQ is run as a service, using Systemd version 240 or lower.
Known issue: https://github.com/systemd/systemd/issues/6559

Solution

Upgrade Systemd to version 241 or newer

or

Set LimitNOFILE in hivemq.service file to a specific value (i.e. 1000000)