HiveMQ Kafka Extension - Cluster Authorization Failed / RANGER-3809

Owned by Aaron Franz
Aug 01, 2023
3 min read

When upgrading from some earlier versions of HiveMQ, the implementation of kafka-client within the HiveMQ Kafka Extension may update to version 3.0+. When this occurs, some Ranger implementations may not automatically handle the change to producer configuration defaults, enabling idempotent write as defined within the associated Apache KIP.

 

This is typically presented by the following error within the HiveMQ Broker logs, available via the log file at /HiveMQ/logs/hivemq.log

org.apache.kafka.common.errors.ClusterAuthorizationException: Cluster authorization failed.

 

This corresponds to an error offered by Apache Kafka, appearing as follows :

RANGER-3809: Idempotent Kafka producer fails to initialize due to an authorization failure

 

In order to correct this, permissions within Apache Ranger must be updated for Kafka Producers to utilize idempotent write.

 

How to Configure Ranger Permissions

Please note that, based on configuration details, the exact names of plugins and permissions may vary, as these can be customized during deployment. Ranger can also be deployed through a few different systems, so exact methods of configuration/Ranger Kafka Plugin access may vary.

  • Log into the Ranger administrator Web UI. This is typically http://{hostname_of_ranger_container}:6080

  • From the home page, select the component plug-in name in the Kafka selection area.

  • Check to see if a ‘Cluster’ option exists. This is typically along the right-side of the Apache Ranger UI. If so :

    • Enter the relevant cluster name and select the cluster.

    • In the Allow Conditions area, select the HiveMQ/MQTT associated user from the Select User drop-down list.

    • Click Add Permissions and select Idempotent Write.

  • Otherwise, click the ‘Add a new Policy’ button.

    • Review if cluster options are now available for the user to edit permissions.

    • If so, enable Idempotent Write for the associated user/usergroup as noted above.

 

How to Add an Internal Config Option (Workaround)

As an optional work-around, in the event that the Ranger permissions cannot be immediately updated, the following can be performed.

Note that this is not a recommended implementation for a production environment, and the associated Ranger permission errors should be resolved for a permanent solution.

  • Navigate to /HiveMQ/Extensions/hivemq-kafka-extension

  • Open kafka-configuration.xml

  • Within the <kafka-configuration> tag, but outside of the other existing tags, add the following :

    <internal> <option> <key>kafka.producer.override.enable.idempotence</key> <value>false</value> </option> </internal>

  • Here is an example configuration :