Reappearing "jetbrains.exodus.ExodusException: Failed to set writable" Error Messages

Overview

This article describes how to identify and resolve recurring {{jetbrains.exodus.ExodusException: Failed to set writable}} error messages in HiveMQ Self-Managed deployments.

Symptoms

• Intermittent error messages in HiveMQ logs: {{jetbrains.exodus.ExodusException: Failed to set writable}}

• Errors occur during persistence garbage collection operations

• Errors typically relate to the {{client_session_subscriptions}} persistence store

Root Cause

The error occurs when HiveMQ's persistence layer attempts to modify file attributes during internal cleanup operations, but lacks the necessary permissions. This is caused by inconsistent file ownership in the HiveMQ persistence directory.

Specifically:

• Persistence files are owned by a different user (often {{root}}) than the user running the HiveMQ broker process

• The HiveMQ garbage collector needs to call {{setWritable()}} on persistence files before deletion

• In Linux/Unix systems, only the file owner can change file permissions, even if group permissions allow read/write/execute access

Diagnosis Steps

1. Verify HiveMQ Process User

Identify which user is running the HiveMQ broker process

2. Check Persistence File Ownership

Examine the ownership of files in the persistence directory:

Look for files where the owner differs from the HiveMQ process user.

3. Review HiveMQ Logs

Check for the specific error pattern in HiveMQ log files located in the log directory.

Resolution Steps

Step 1: Correct File Ownership

Recursively change ownership of the entire persistence directory to match the HiveMQ process user:

Example:

Step 2: Verify Directory Permissions

Ensure the HiveMQ data directory has appropriate read, write, and execute permissions for the HiveMQ process user.

Step 3: Restart HiveMQ

After correcting ownership, restart the HiveMQ broker to ensure clean operation.

Step 4: Monitor for Recurrence

Monitor HiveMQ logs for several days to confirm the error no longer appears.

Prevention Recommendations

To prevent this issue from recurring:

1. Consistent User Execution: Always start HiveMQ with the same dedicated non-root user account. Verify startup scripts and systemd service files use the correct user.

2. Deployment Script Review: Ensure deployment, backup, restore, and maintenance scripts do not create or modify persistence files as root or any user other than the HiveMQ process user.

3. Automated Ownership Verification: Include ownership verification steps in deployment automation to ensure all files in the data/persistence directory are owned by the HiveMQ process user.

4. Access Control: Restrict administrative access to the persistence directory to prevent accidental file creation or modification with incorrect ownership.

Additional Notes

• HiveMQ can run as root or as a dedicated service user, but all persistence files must be consistently owned by whichever user runs the broker process.

• Even when group permissions appear correct, Linux/Unix file permission semantics require the file owner to match the process user for permission modification operations.

• The HiveMQ data directory structure requires read, write, and execute permissions for proper operation.