# PXC Operator failed to update service with new annotations on the haproxy created service

**URL:** <https://forums.percona.com/t/pxc-operator-failed-to-update-service-with-new-annotations-on-the-haproxy-created-service/34582>\
**Category:** Percona Operator for MySQL\
**Created:** [October 24, 2024, 1:49pm UTC](https://forums.percona.com/t/pxc-operator-failed-to-update-service-with-new-annotations-on-the-haproxy-created-service/34582 "2024-10-24T13:49:56Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Gerwin\_van\_de\_Steeg](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/gerwin_van_de_steeg/32/3715_2.png) [@Gerwin\_van\_de\_Steeg](https://forums.percona.com/u/Gerwin_van_de_Steeg)\
**Post date:** [October 24, 2024, 1:49pm UTC](https://forums.percona.com/t/pxc-operator-failed-to-update-service-with-new-annotations-on-the-haproxy-created-service/34582/1 "2024-10-24T13:49:56Z")

</div>

## Description:

Operator doesn’t synchronise service annotations correctly

The section from the perconaXtraDBCluster chart looks like

```auto
  crVersion: 1.15.0
  enableCRValidationWebhook: false
  haproxy:
    affinity:
      antiAffinityTopologyKey: kubernetes.io/hostname
    annotations:
      prometheus.io/port: "8404"
      prometheus.io/scrape: "true"
    enabled: true
    exposePrimary:
      annotations:
        external-dns.alpha.kubernetes.io/hostname: db-redacted1.prod.redaced.local,db-redacted2.prod.redacted.local,db-redacted3.prod.redacted.local
        external-dns.alpha.kubernetes.io/ttl: "60"
        kubernetes.io/ingress.class: internal-nginx
        service.beta.kubernetes.io/aws-load-balancer-additional-resource-tags: creation:Tool=flux,project:Name=redaced,Purpose=SQL
        service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp
        service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: "3600"
        service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
        service.beta.kubernetes.io/aws-load-balancer-healthcheck-port: traffic-port
        service.beta.kubernetes.io/aws-load-balancer-healthcheck-protocol: tcp
        service.beta.kubernetes.io/aws-load-balancer-name: nv-internal-haproxy-sql
        service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
        service.beta.kubernetes.io/aws-load-balancer-scheme: internal
        service.beta.kubernetes.io/aws-load-balancer-type: external
      enabled: true
      loadBalancerSourceRanges:
      - 10.0.0.0/8
      type: LoadBalancer

```

The created service looks like

```auto
$ kubectl get svc voipmonitor-db-pxc-db-haproxy -o yaml
apiVersion: v1
kind: Service
metadata:
  annotations:
    external-dns.alpha.kubernetes.io/ttl: "60"
    kubernetes.io/ingress.class: internal-nginx
    percona.com/last-config-hash: redacted
    service.beta.kubernetes.io/aws-load-balancer-additional-resource-tags: creation:Tool=flux,project:Name=redacted,Purpose=SQL
    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp
    service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: "3600"
    service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
    service.beta.kubernetes.io/aws-load-balancer-healthcheck-port: traffic-port
    service.beta.kubernetes.io/aws-load-balancer-healthcheck-protocol: tcp
    service.beta.kubernetes.io/aws-load-balancer-name: nv-internal-voipmonitor-sql
    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
    service.beta.kubernetes.io/aws-load-balancer-scheme: internal
    service.beta.kubernetes.io/aws-load-balancer-type: external

```

This caused the internal DNS entries for these services that are then used by workloads outside the Kubernetes cluster to not get created.

## Steps to Reproduce:

Unsure, the final `external-dns.alpha.kubernetes.io/hostname` was added last, after the resource had already been created with all relevant settings.

## Version:

$ kubectl version  
Client Version: v1.31.1  
Kustomize Version: v5.4.2  
Server Version: v1.28.13-eks-a737599

pxc-operator-1.15.0

## Logs:

```auto
2024-10-25T03:22:15.694Z ERROR Reconciler error {"controller": "pxc-controller", "namespace": "default", "name": "voipmonitor-db-pxc-db", "reconcileID": "dc5e64ff-7518-433c-9311-a27898a57020", "error": "voipmonitor-db-pxc-db-haproxy upgrade error: Service \"voipmonitor-db-pxc-db-haproxy\" is invalid: spec.loadBalancerClass: Invalid value: \"null\": may not change once set", "errorVerbose": "Service \"voipmonitor-db-pxc-db-haproxy\" is invalid: spec.loadBalancerClass: Invalid value: \"null\": may not change once set\nvoipmonitor-db-pxc-db-haproxy upgrade error\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc.(*ReconcilePerconaXtraDBCluster).reconcileHAProxy\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc/controller.go:495\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc.(*ReconcilePerconaXtraDBCluster).Reconcile\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc/controller.go:366\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:222\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1695"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:324
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:261
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:222

```

## Expected Result:

The DNS entries get created by the External DNS setup to create internal DNS entries

## Actual Result:

The service never got the annotations updated to include those new values so External-DNS never picked them up.

## Additional Information:

n/a

---

<div class="post-metadata">

**Author:** ![Gerwin\_van\_de\_Steeg](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/gerwin_van_de_steeg/32/3715_2.png) [@Gerwin\_van\_de\_Steeg](https://forums.percona.com/u/Gerwin_van_de_Steeg)\
**Post date:** [October 25, 2024, 3:27am UTC](https://forums.percona.com/t/pxc-operator-failed-to-update-service-with-new-annotations-on-the-haproxy-created-service/34582/2 "2024-10-25T03:27:22Z")

</div>

looks like the controller is trying to edit the Service and encountering an issue altering an immutable field.

```auto
"error": "voipmonitor-db-pxc-db-haproxy upgrade error: Service \"voipmonitor-db-pxc-db-haproxy\" is invalid: spec.loadBalancerClass: Invalid value: \"null\": may not change once set", "errorVerbose": "Service \"voipmonitor-db-pxc-db-haproxy\" is invalid: spec.loadBalancerClass: Invalid value: \"null\": may not change once set\nvoipmonitor-db-pxc-db-haproxy upgrade error\n

```

---

<div class="post-metadata">

**Author:** ![Sergey\_Pronin](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/sergey_pronin/32/14887_2.png) [@Sergey\_Pronin](https://forums.percona.com/u/Sergey_Pronin)\
**Post date:** [October 28, 2024, 8:33am UTC](https://forums.percona.com/t/pxc-operator-failed-to-update-service-with-new-annotations-on-the-haproxy-created-service/34582/3 "2024-10-28T08:33:14Z")

</div>

Hey @Gerwin_van_de_Steeg

Seems like a bug. I have a hypothesis that these two can be triggering it:  
[https://perconadev.atlassian.net/browse/K8SPXC-1267](https://perconadev.atlassian.net/browse/K8SPXC-1267)  
[https://perconadev.atlassian.net/browse/K8SPXC-1061](https://perconadev.atlassian.net/browse/K8SPXC-1061)

Checking with the team. I assume you have not created any bugs yet, right?

---

<div class="post-metadata">

**Author:** ![Gerwin\_van\_de\_Steeg](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/gerwin_van_de_steeg/32/3715_2.png) [@Gerwin\_van\_de\_Steeg](https://forums.percona.com/u/Gerwin_van_de_Steeg)\
**Post date:** [October 28, 2024, 8:07pm UTC](https://forums.percona.com/t/pxc-operator-failed-to-update-service-with-new-annotations-on-the-haproxy-created-service/34582/4 "2024-10-28T20:07:31Z")

</div>

Nope, no bugs created in JIRA from me.

---

<div class="post-metadata">

**Author:** ![Gerwin\_van\_de\_Steeg](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/gerwin_van_de_steeg/32/3715_2.png) [@Gerwin\_van\_de\_Steeg](https://forums.percona.com/u/Gerwin_van_de_Steeg)\
**Post date:** [October 30, 2024, 4:23am UTC](https://forums.percona.com/t/pxc-operator-failed-to-update-service-with-new-annotations-on-the-haproxy-created-service/34582/5 "2024-10-30T04:23:10Z")

</div>

Sergey,

This doesn’t look like an issue with the two tickets you mentioned but related to the `loadBalancerClass` the service which worked has:

```auto
$ kubectl get svc voipmonitor-db-pxc-db-haproxy -o jsonpath='{.spec.loadBalancerClass}'
service.k8s.aws/nlb

```

The services that are failing to update have:

```auto
$ kubectl get svc mysql-pxc-db-haproxy -o jsonpath='{.spec.loadBalancerClass}'

```

This seems to be related to that immutable field.

---

<div class="post-metadata">

**Author:** ![Gerwin\_van\_de\_Steeg](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/gerwin_van_de_steeg/32/3715_2.png) [@Gerwin\_van\_de\_Steeg](https://forums.percona.com/u/Gerwin_van_de_Steeg)\
**Post date:** [December 9, 2024, 9:19pm UTC](https://forums.percona.com/t/pxc-operator-failed-to-update-service-with-new-annotations-on-the-haproxy-created-service/34582/6 "2024-12-09T21:19:13Z")

</div>

@Sergey_Pronin any update here?

This is also related to [Operator fails reconciling database spec with LoadBalancerSourceRanges specified · Issue #1899 · percona/percona-xtradb-cluster-operator · GitHub](https://github.com/percona/percona-xtradb-cluster-operator/issues/1899)  
We’re about to go into the annual shut down and these bugs are preventing the backups from getting created.