Hello everyone,
We have deployed a percona mysql operator 1.9 on K8S and we added an HPA on the cluster. Everything works fine however we have noticed that almost everytime a scale down happens haproxy gives an error and restarts. We see this on logs on pxc-monit on ha proxy:
+ cat
+ IFS='
'
+ echo 'server db-cluster-pxc-0 db-cluster-pxc-0.db-cluster-pxc.-m2.svc.cluster.local:33060 check inter 10000 rise 1 fall 2 weight 1 on-marked-up shutdown-backup-sessions
server db-cluster-pxc-3 db-cluster-pxc-3.db-cluster-pxc.-m2.svc.cluster.local:33060 check inter 10000 rise 1 fall 2 weight 1 backup
server db-cluster-pxc-2 db-cluster-pxc-2.db-cluster-pxc.-m2.svc.cluster.local:33060 check inter 10000 rise 1 fall 2 weight 1 backup
server db-cluster-pxc-1 db-cluster-pxc-1.db-cluster-pxc.-m2.svc.cluster.local:33060 check inter 10000 rise 1 fall 2 weight 1 backup'
+ SOCKET=/etc/haproxy/pxc/haproxy.sock
+ path_to_custom_global_cnf=/etc/haproxy-custom
+ '[' -f /etc/haproxy-custom/haproxy-global.cfg ']'
+ '[' -f /etc/haproxy-custom/haproxy-global.cfg -a -z '' ']'
+ haproxy -c -f /etc/haproxy/haproxy-global.cfg -f /etc/haproxy/pxc/haproxy.cfg
[NOTICE] 312/170344 (402) : haproxy version is 2.3.10
[NOTICE] 312/170344 (402) : path to executable is /usr/sbin/haproxy
[ALERT] 312/170344 (402) : parsing [/etc/haproxy/pxc/haproxy.cfg:8] : 'server db-cluster-pxc-3' : could not resolve address 'db-cluster-pxc-3.db-cluster-pxc.-m2.svc.cluster.local'.
[ALERT] 312/170344 (402) : parsing [/etc/haproxy/pxc/haproxy.cfg:18] : 'server db-cluster-pxc-3' : could not resolve address 'db-cluster-pxc-3.db-cluster-pxc.-m2.svc.cluster.local'.
[ALERT] 312/170344 (402) : parsing [/etc/haproxy/pxc/haproxy.cfg:30] : 'server db-cluster-pxc-3' : could not resolve address 'db-cluster-pxc-3.db-cluster-pxc.-m2.svc.cluster.local'.
[ALERT] 312/170344 (402) : parsing [/etc/haproxy/pxc/haproxy.cfg:38] : 'server db-cluster-pxc-3' : could not resolve address 'db-cluster-pxc-3.db-cluster-pxc.-m2.svc.cluster.local'.
[ALERT] 312/170344 (402) : Failed to initialize server(s) addr.
It seems that haproxy doesn’t update correctly when scale down happens. On the operator logs it could be related it says the following:
{"level":"error","ts":1636477422.5753634,"logger":"controller","msg":"Reconciler error","controller":"perconaxtradbcluster-controller","name":"db-cluster","namespace":"-m2","error":"pxc upgrade error: update error: Operation cannot be fulfilled on statefulsets.apps \"db-cluster-pxc\": the object has been modified; please apply your changes to the latest version and try again","errorVerbose":"Operation cannot be fulfilled on statefulsets.apps \"db-cluster-pxc\": the object has been modified; please apply your changes to the latest version and try again\nupdate error\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc.(*ReconcilePerconaXtraDBCluster).updatePod\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc/upgrade.go:201\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc.(*ReconcilePerconaXtraDBCluster).Reconcile\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc/controller.go:339\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:235\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:209\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:188\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371\npxc upgrade error\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc.(*ReconcilePerconaXtraDBCluster).Reconcile\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc/controller.go:341\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:235\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:209\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:188\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:237\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:209\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:188\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90"}
Is there something we are missing? Any ideas?
Thanks in advance!