Operator repeatedly shutdown/restart every 2 hours

Question

What could be the reason that Operator repeatedly shutdown/restart a node after exactly 2 hours?

Answer

The named “mqtt” port must be on a port that exists. In case when the “mqtt” port is set to 1883 but not configured in the listenerConfiguration, it will prevent the liveness probe from functioning and the node will subsequently fail.

The liveness probe auto-configures to the port named “mqtt”. If the “mqtt” port is 1883, it will check 1883 tcp socket. If the port 1883 is not configured in the listenerConfiguration, then this port will not be open on the pods and the liveness probe will fail.

To fix, either make sure that the port named “mqtt” is configured in the listenerConfiguration or pick a port that is already configured in the listenerConfiguration and rename it to "mqtt".

The following in the values.yaml will expose the port named "mqtt" with port number 1883:

 hivemq: 
  ports:
  - name: "mqtt"
    port: 1883
    expose: true

The matching listenerConfiguration in the config.xml:

<hivemq>

    <listeners>
        <tcp-listener>
            <port>1883</port>
            <bind-address>0.0.0.0</bind-address>
        </tcp-listener>
    </listeners>
    
</hivemq>