HiveMQ Kubernetes Operator UPGRADE FAILED: post-upgrade hooks failed: job failed
- Daria Samkova
Problem
HiveMQ Kubernetes Operator has options to be installed along with Prometheus and Grafana:
helm upgrade $releaseName --install 'hivemq/hivemq-operator' \
--set monitoring.enabled=true \
--set monitoring.dedicated=true This includes the dependency from kube-prometheus-stack into the hivemq-operator chart.
hivemq-operator/Chart.yaml:
...
dependencies:
- condition: monitoring.dedicated, monitoring.enabled
name: kube-prometheus-stack
repository: https://prometheus-community.github.io/helm-charts
version: 14.*
...The kube-prometheus-stack version 14.* is specifying admissionWebhooks = enabled and admission-patch job to use the image jettech/kube-webhook-certgen:v1.5.0
kube-prometheus-stack/values.yaml:
...
prometheusOperator:
enabled: true
...
admissionWebhooks:
failurePolicy: Fail
enabled: true
...
patch:
enabled: true
image:
repository: jettech/kube-webhook-certgen
tag: v1.5.0
sha: ""
pullPolicy: IfNotPresent
When installing on Kubernetes cluster v.1.23 this job fails. Job’s name is ${releaseName}-kube-prometheu-admission-patch.
Solution
Use a different image for the kube-prometheus-admission-patch Job, one that does not fail:
helm upgrade $releaseName --install hivemq/hivemq-operator \
--set monitoring.enabled=true \
--set monitoring.dedicated=true \
--set kube-prometheus-stack.prometheusOperator.admissionWebhooks.patch.image.repository=k8s.gcr.io/ingress-nginx/kube-webhook-certgen \
--set kube-prometheus-stack.prometheusOperator.admissionWebhooks.patch.image.tag=v1.1.1
Alternatively, edit the hivemq-operator/values.yaml file and add the fixed values:
hivemq-operator/values.yaml → values-monitoring-fix.yaml:
global:
...
operator:
...
hivemq:
...
monitoring:
enabled: true
dedicated: true
kube-prometheus-stack:
prometheusOperator:
admissionWebhooks:
patch:
image:
repository: registry.k8s.io/ingress-nginx/kube-webhook-certgen
tag: v1.1.1And install with values.yaml
helm upgrade $releaseName --install hivemq/hivemq-operator \
--values values-monitoring-fix.yamlIf there is “No data” in the Grafana, refer to the following article for further troubleshooting: No data in Grafana after installing HiveMQ Kubernetes Operator with monitoring