Cannot create new topic in Confluent Cloud Kafka cluster

Owned by Daria Samkova, created
May 02, 2024
1 min read

 Problem

I created an mqtt-to-kafka-transformer, which should dynamically create a new topic in Kafka. However, this fails:

WARN - [Producer clientId=hivemq-kafka-extension-producer-1] Error while fetching metadata with correlation id 1 : {my-new-topic=UNKNOWN_TOPIC_OR_PARTITION}

 Solution

  1. Go to the Confluent Cloud Kafka cluster and verify how many partitions the topic there has.

  2. If it is different from the default 3, then adjust this setting in the kafka-extension configuration file

    <kafka-configuration> ... ... <internal> <option> <key>kafka.topic.default.replication-factor</key> <value>3</value> </option> <option> <key>kafka.topic.default.partitions</key> <value>6</value> </option> </internal> </kafka-configuration>

  3. Restart the kafka-extension and test again.

 

If the error persists, copy the error message and exception stack trace and contact HiveMQ Support for further investigation.