HiveMQ Kubernetes Operator UPGRADE FAILED: post-upgrade hooks failed: job failed
- Daria Samkova
Problem
HiveMQ Kubernetes Operator has options to be installed along with Prometheus and Grafana:
helm upgrade $releaseName --install 'hivemq/hivemq-operator' \
--set monitoring.enabled=true \
--set monitoring.dedicated=true This includes the dependency from kube-prometheus-stack into the hivemq-operator chart.
hivemq-operator/Chart.yaml:
...
dependencies:
- condition: monitoring.dedicated, monitoring.enabled
name: kube-prometheus-stack
repository: https://prometheus-community.github.io/helm-charts
version: 14.*
...The kube-prometheus-stack version 14.* is specifying admissionWebhooks = enabled and admission-patch job to use the image jettech/kube-webhook-certgen:v1.5.0
kube-prometheus-stack/values.yaml:
...
prometheusOperator:
enabled: true
...
admissionWebhooks:
failurePolicy: Fail
enabled: true
...
patch:
enabled: true
image:
repository: jettech/kube-webhook-certgen
tag: v1.5.0
sha: ""
pullPolicy: IfNotPresent
When installing on Kubernetes cluster v.1.23 this job fails. Job’s name is ${releaseName}-kube-prometheu-admission-patch.
Solution
Use a different image for the kube-prometheus-admission-patch Job, one that does not fail:
Alternatively, edit the hivemq-operator/values.yaml file and add the fixed values:
hivemq-operator/values.yaml → values-monitoring-fix.yaml:
And install with values.yaml
If there is “No data” in the Grafana, refer to the following article for further troubleshooting: No data in Grafana after installing HiveMQ Kubernetes Operator with monitoring