Description:
Operator doesn’t synchronise service annotations correctly
The section from the perconaXtraDBCluster chart looks like
crVersion: 1.15.0
enableCRValidationWebhook: false
haproxy:
affinity:
antiAffinityTopologyKey: kubernetes.io/hostname
annotations:
prometheus.io/port: "8404"
prometheus.io/scrape: "true"
enabled: true
exposePrimary:
annotations:
external-dns.alpha.kubernetes.io/hostname: db-redacted1.prod.redaced.local,db-redacted2.prod.redacted.local,db-redacted3.prod.redacted.local
external-dns.alpha.kubernetes.io/ttl: "60"
kubernetes.io/ingress.class: internal-nginx
service.beta.kubernetes.io/aws-load-balancer-additional-resource-tags: creation:Tool=flux,project:Name=redaced,Purpose=SQL
service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp
service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: "3600"
service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
service.beta.kubernetes.io/aws-load-balancer-healthcheck-port: traffic-port
service.beta.kubernetes.io/aws-load-balancer-healthcheck-protocol: tcp
service.beta.kubernetes.io/aws-load-balancer-name: nv-internal-haproxy-sql
service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
service.beta.kubernetes.io/aws-load-balancer-scheme: internal
service.beta.kubernetes.io/aws-load-balancer-type: external
enabled: true
loadBalancerSourceRanges:
- 10.0.0.0/8
type: LoadBalancer
The created service looks like
$ kubectl get svc voipmonitor-db-pxc-db-haproxy -o yaml
apiVersion: v1
kind: Service
metadata:
annotations:
external-dns.alpha.kubernetes.io/ttl: "60"
kubernetes.io/ingress.class: internal-nginx
percona.com/last-config-hash: redacted
service.beta.kubernetes.io/aws-load-balancer-additional-resource-tags: creation:Tool=flux,project:Name=redacted,Purpose=SQL
service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp
service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: "3600"
service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
service.beta.kubernetes.io/aws-load-balancer-healthcheck-port: traffic-port
service.beta.kubernetes.io/aws-load-balancer-healthcheck-protocol: tcp
service.beta.kubernetes.io/aws-load-balancer-name: nv-internal-voipmonitor-sql
service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
service.beta.kubernetes.io/aws-load-balancer-scheme: internal
service.beta.kubernetes.io/aws-load-balancer-type: external
This caused the internal DNS entries for these services that are then used by workloads outside the Kubernetes cluster to not get created.
Steps to Reproduce:
Unsure, the final external-dns.alpha.kubernetes.io/hostname was added last, after the resource had already been created with all relevant settings.
Version:
$ kubectl version
Client Version: v1.31.1
Kustomize Version: v5.4.2
Server Version: v1.28.13-eks-a737599
pxc-operator-1.15.0
Logs:
2024-10-25T03:22:15.694Z ERROR Reconciler error {"controller": "pxc-controller", "namespace": "default", "name": "voipmonitor-db-pxc-db", "reconcileID": "dc5e64ff-7518-433c-9311-a27898a57020", "error": "voipmonitor-db-pxc-db-haproxy upgrade error: Service \"voipmonitor-db-pxc-db-haproxy\" is invalid: spec.loadBalancerClass: Invalid value: \"null\": may not change once set", "errorVerbose": "Service \"voipmonitor-db-pxc-db-haproxy\" is invalid: spec.loadBalancerClass: Invalid value: \"null\": may not change once set\nvoipmonitor-db-pxc-db-haproxy upgrade error\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc.(*ReconcilePerconaXtraDBCluster).reconcileHAProxy\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc/controller.go:495\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc.(*ReconcilePerconaXtraDBCluster).Reconcile\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc/controller.go:366\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:222\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1695"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:324
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:261
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:222
Expected Result:
The DNS entries get created by the External DNS setup to create internal DNS entries
Actual Result:
The service never got the annotations updated to include those new values so External-DNS never picked them up.
Additional Information:
n/a