Data retention for a persistent session seems to be limited to 1,000 records

Explanation

Clients will, by default, utilize the globally configured maximum queue size of 1000 messages: max-queue-size. To check this, you can navigate to the subscriber client in the HiveMQ Control Center's Clients list. It will display both the current and maximum queue size for the client. You can observe that the "Maximum" value is set to 1,000:

When you publish to the topic while the subscriber is offline, the queue gradually increases up to the limit of 1000. Beyond this threshold, messages will start getting dropped.

Solution

You can use Shared Subscriptions, which you can find more information about here: https://docs.hivemq.com/hivemq/4.18/user-guide/shared-subscriptions.html#structure

A shared subscription has a significantly higher queue limit of 500,000. By subscribing to the topic filter e.g. "$share/demo1/Test/#" instead of "Test/#," you can activate a shared subscription with a Maximum Message Queue Size of 500,000.

With shared subscriptions, the broker can queue many more messages, effectively addressing this issue.

Please reach out to your HiveMQ support team in case further help is required.