Cluster status and backups not working

MySQL & MariaDB Percona Operator for MySQL
1

Greetings,

Is this expected to not show any status?

❯ k get pxc mysql
NAME    ENDPOINT   STATUS   PXC   PROXYSQL   HAPROXY   AGE
mysql                                                  3h18m

Cluster seems fine otherwise

❯ k get pods --selector app.kubernetes.io/instance=mysql
NAME              READY   STATUS    RESTARTS   AGE
mysql-haproxy-0   2/2     Running   0          3h19m
mysql-haproxy-1   2/2     Running   0          3h16m
mysql-haproxy-2   2/2     Running   0          3h16m
mysql-pxc-0       3/3     Running   0          3h19m
mysql-pxc-1       3/3     Running   0          3h17m
mysql-pxc-2       3/3     Running   0          3h13m

❯ k get pxc mysql -o yaml
apiVersion: pxc.percona.com/v1
kind: PerconaXtraDBCluster
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"pxc.percona.com/v1-9-0","kind":"PerconaXtraDBCluster"}
    meta.helm.sh/release-name: app
    meta.helm.sh/release-namespace: men
  creationTimestamp: "2021-10-27T11:44:40Z"
  finalizers:
  - delete-pxc-pods-in-order
  - delete-proxysql-pvc
  - delete-pxc-pvc
  generation: 2
  labels:
    app.kubernetes.io/instance: app
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: mysql
    app.kubernetes.io/version: 1.9.0
    helm.sh/chart: pxc-db-1.9.1
  name: mysql
  namespace: men
  resourceVersion: "290700413"
  uid: fedd4665-5166-4e8d-8fbd-27ef6a3fc457
spec:
  allowUnsafeConfigurations: true
  backup:
    image: percona/percona-xtradb-cluster-operator:1.9.0-pxc5.7-backup
    imagePullPolicy: Always
    pitr:
      enabled: true
      storageName: scw-poc-percona-backups
      timeBetweenUploads: 60
    schedule:
    - keep: 5
      name: daily-backup
      schedule: 0 0 * * *
      storageName: scw-poc-percona-backups
    storages:
      fs-pvc:
        podSecurityContext:
          fsGroup: 1001
          supplementalGroups:
          - 1001
        s3:
          bucket: ""
          credentialsSecret: ""
        type: filesystem
        volume:
          persistentVolumeClaim:
            accessModes:
            - ReadWriteOnce
            resources:
              requests:
                storage: 6Gi
      scw-poc-percona-backups:
        podSecurityContext:
          fsGroup: 1001
          supplementalGroups:
          - 1001
        s3:
          bucket: XXXXXXXXXXXXXXXXXXXXXx
          credentialsSecret: mysql-s3-XXXXXXXXXXXXXXXXXXXXXx
          endpointUrl: https://s3.fr-par.scw.cloud
          region: fr-par
        type: s3
  crVersion: 1.9.0
  enableCRValidationWebhook: false
  haproxy:
    affinity:
      antiAffinityTopologyKey: kubernetes.io/hostname
    annotations:
      percona.com/configuration-hash: d41d8cd98f00b204e9800998ecf8427e
    enabled: true
    envVarsSecret: mysql-env-vars-haproxy
    gracePeriod: 30
    image: percona/percona-xtradb-cluster-operator:1.9.0-haproxy
    imagePullPolicy: Always
    livenessProbes:
      exec:
        command:
        - /usr/local/bin/liveness-check.sh
      failureThreshold: 4
      initialDelaySeconds: 60
      periodSeconds: 30
      successThreshold: 1
      timeoutSeconds: 5
    nodeSelector:
      andjaro.io/system: "true"
    podDisruptionBudget:
      maxUnavailable: 1
    readinessProbes:
      exec:
        command:
        - /usr/local/bin/readiness-check.sh
      failureThreshold: 3
      initialDelaySeconds: 15
      periodSeconds: 5
      successThreshold: 1
      timeoutSeconds: 1
    resources:
      limits: {}
      requests:
        cpu: 600m
        memory: 1G
    serviceAccountName: default
    sidecarResources:
      limits: {}
      requests: {}
    size: 3
    tolerations:
    - effect: NoSchedule
      key: k8s.scaleway.com/dedicated
      operator: Equal
      value: system
    volumeSpec:
      emptyDir: {}
  logCollectorSecretName: mysql-log-collector
  logcollector:
    enabled: true
    image: percona/percona-xtradb-cluster-operator:1.9.0-logcollector
    imagePullPolicy: Always
    resources:
      limits: {}
      requests: {}
  platform: kubernetes
  pmm:
    resources:
      limits: {}
      requests:
        cpu: 600m
        memory: 1G
  proxysql:
    livenessProbes: {}
    podSecurityContext:
      fsGroup: 1001
      supplementalGroups:
      - 1001
    readinessProbes: {}
  pxc:
    affinity:
      antiAffinityTopologyKey: kubernetes.io/hostname
    annotations:
      percona.com/configuration-hash: d41d8cd98f00b204e9800998ecf8427e
    autoRecovery: true
    envVarsSecret: mysql-env-vars-pxc
    expose: {}
    gracePeriod: 600
    image: percona/percona-xtradb-cluster:5.7.34-31.51
    imagePullPolicy: Always
    livenessDelaySec: 300
    livenessProbes:
      exec:
        command:
        - /var/lib/mysql/liveness-check.sh
      failureThreshold: 3
      initialDelaySeconds: 300
      periodSeconds: 10
      successThreshold: 1
      timeoutSeconds: 5
    nodeSelector:
      andjaro.io/system: "true"
    podDisruptionBudget:
      maxUnavailable: 1
    podSecurityContext:
      fsGroup: 1001
      supplementalGroups:
      - 1001
    readinessDelaySec: 15
    readinessProbes:
      exec:
        command:
        - /var/lib/mysql/readiness-check.sh
      failureThreshold: 5
      initialDelaySeconds: 15
      periodSeconds: 30
      successThreshold: 1
      timeoutSeconds: 15
    resources:
      limits: {}
      requests:
        cpu: 600m
        memory: 1G
    serviceAccountName: default
    sidecarResources:
      limits: {}
      requests: {}
    size: 3
    sslInternalSecretName: mysql-ssl-internal
    sslSecretName: mysql-ssl
    tolerations:
    - effect: NoSchedule
      key: k8s.scaleway.com/dedicated
      operator: Equal
      value: system
    vaultSecretName: mysql-vault
    volumeSpec:
      persistentVolumeClaim:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 10Gi
        storageClassName: scw-bssd
  secretsName: mysql
  updateStrategy: SmartUpdate
  upgradeOptions:
    apply: Never
    schedule: 0 4 * * *
    versionServiceEndpoint: https://check.percona.com
  vaultSecretName: mysql-vault

Also this is affecting backups - error from operator

{"level":"error","ts":1635347117.5467057,"logger":"controller","msg":"Reconciler error","controller":"perconaxtradbclusterbackup-controller","name":"cron-mysql-scw-poc-percona-20211027000-372f8","namespace":"men","error":"failed to run backup: there are no ready PXC nodes","errorVerbose":"there are no ready PXC nodes\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/apis/pxc/v1.(*PerconaXtraDBCluster).CanBackup\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/apis/pxc/v1/pxc_types.go:1086\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).Reconcile\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:177\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:235\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:209\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:188\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371\nfailed to run backup\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).Reconcile\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:178\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:235\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:209\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:188\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:237\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:209\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:188\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90"}

Why status is not populated? Am I missing something?#

Thanks

1 Like
2

Hi @pavloos,

We need to know k8s version and steps to reproduce this issue.

1 Like
3

Thanks @Slava_Sarzhan

  • kubernetes v1.22.2 (Scaleway Kosmos cluster)
  • operator installed in db namespace with helm chart and watchAllNamespaces: true (chart version: 1.9.1)
  • PerconaXtraDBCluster installed in different namespace as seen in the yaml provider

I tried with haproxy and proxysql, the same result.

4

@pavloos 1.9 does not fully support k8s 1.22 and it will be fixed in 1.10. Could you please have a look at the following post svc/${clustername}-pxc-unready (you can find the workaround here). Or you can use k8s <= 1.21.

1 Like
5

It is running with the fix applied manually yet the status is not updated so it is different issue unless there are other incompatibilities apart from publishNotReadyAddresses missing. Cluster itself is operational and I can connect to.

I will try with 1.21 and let you know.

In the meantime if I can provide any other information please do let me know.

1 Like
6

I confirm that in kubernetes v 1.21.5 it properly populates status of the pxc object

❯ k get pxc mysql
NAME    ENDPOINT            STATUS   PXC   PROXYSQL   HAPROXY   AGE
mysql   mysql-haproxy.men   ready    2                2         15m

Although the service still needs to be fixed with publishNotRreadyAddresses: true.

I will wait for the 1.10 release then! Thanks for your help

2 Likes