PITR restore fails when backup is on FS and binlogs on S3 minio

Description:

It is not possible to do PITR restore when full database backup is on filesystem and pitr pod holding binlogs on S3 minio.
Restore ends with error and “unknown storage type”

Restore normally works if:

  • it is not PITR
  • it is PITR but both backup and binlogs are on S3 minio storage

Steps to Reproduce:

  1. configure backup on filesystem
  2. enable pitr and configure to S3 minio
  3. do manual backup on filesystem or wait conjob to make one
  4. do some transactions in the database to verify
  5. verify the transaction is in binlog on S3 minio
  6. run PITR restore

Version:

oc version
Client Version: 4.12.28
Kustomize Version: v4.5.7
Server Version: 4.12.28
Kubernetes Version: v1.25.11+1485cc9
+-------------+
| @@version   |
+-------------+
| 8.0.32-24.2 |
+-------------+
percona-xtradb-cluster-operator:1.13.0-pxc8.0-backup-pxb8.0.32

Logs:

none

Expected Result:

Database to be restored to latest available transaction in latest available binlog

Actual Result:

The result is incomplete restore, 3 haproxy pods started but only 1 pxc pod.

NAME                                          READY   STATUS      RESTARTS   AGE
ope04-haproxy-0                               2/2     Running     0          24h
ope04-haproxy-1                               2/2     Running     0          23h
ope04-haproxy-2                               2/2     Running     0          23h
ope04-pitr-5dcb96459-gdptq                    1/1     Running     0          23h
ope04-pxc-0                                   3/3     Running     0          23h
ope04-pxc-operator-7b9677c5b9-bsq88           1/1     Running     0          8d
pxc-minio-5555f54887-k2rt6                    1/1     Running     0          7d22h

All attempts to scale up remaining two pxc pods leads pods to be in pending state

status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2024-04-24T20:07:46Z"
    message: '0/6 nodes are available: 3 node(s) didn''t find available persistent
      volumes to bind, 3 node(s) had untolerated taint {node-role.kubernetes.io/master:
      }. preemption: 0/6 nodes are available: 6 Preemption is not helpful for scheduling.'
    reason: Unschedulable
    status: "False"
    type: PodScheduled
  phase: Pending
  qosClass: Burstable

In order to scale up pxc pods, it was necessary to recreate PVs on which those pods are running

Additional Information:

PXC operator backup yaml

      pitr:
        enabled: true
        resources: {}
        storageName: bcp-s3
        timeBetweenUploads: 60
      schedule:
      - keep: 3
        name: daily-backup
        schedule: 0 0 * * *
        storageName: fs-pvc
      serviceAccountName: percona-xtradb-cluster-operator
      storages:
        bcp-s3:
          podSecurityContext:
            supplementalGroups:
            - 1001
          resources: {}
          s3:
            bucket: pitr-backup
            credentialsSecret: pxc-minio
            endpointUrl: http://pxc-minio:9000
          type: s3
        fs-pvc:
          podSecurityContext:
            fsGroup: 1001
            supplementalGroups:
            - 1001
            - 1002
            - 1003
          resources: {}
          type: filesystem
          volume:
            persistentVolumeClaim:
              accessModes:
              - ReadWriteOnce
              resources:
                requests:
                  storage: 50Gi
              storageClassName: rook-minio

Backups

k get pxc-backup
NAME                                 CLUSTER   STORAGE   DESTINATION                                       STATUS      COMPLETED   AGE
cron-ope04-fs-pvc-2024550037-372f8   ope04     fs-pvc    pvc/xb-cron-ope04-fs-pvc-2024550037-372f8         Succeeded   2d20h       2d20h
cron-ope04-fs-pvc-2024560037-372f8   ope04     fs-pvc    pvc/xb-cron-ope04-fs-pvc-2024560037-372f8         Succeeded   44h         44h
cron-ope04-fs-pvc-2024570052-372f8   ope04     fs-pvc    pvc/xb-cron-ope04-fs-pvc-2024570052-372f8         Succeeded   20h         20h
s3testbackup1                        ope04     bcp-s3    s3://pitr-backup/ope04-2024-05-06-16:10:53-full   Succeeded   28h         28h

Storage

k get pxc-backup
NAME                                 CLUSTER   STORAGE   DESTINATION                                       STATUS      COMPLETED   AGE
cron-ope04-fs-pvc-2024550037-372f8   ope04     fs-pvc    pvc/xb-cron-ope04-fs-pvc-2024550037-372f8         Succeeded   2d20h       2d20h
cron-ope04-fs-pvc-2024560037-372f8   ope04     fs-pvc    pvc/xb-cron-ope04-fs-pvc-2024560037-372f8         Succeeded   44h         44h
cron-ope04-fs-pvc-2024570052-372f8   ope04     fs-pvc    pvc/xb-cron-ope04-fs-pvc-2024570052-372f8         Succeeded   20h         20h
s3testbackup1                        ope04     bcp-s3    s3://pitr-backup/ope04-2024-05-06-16:10:53-full   Succeeded   28h         28h

Restore

Spec:
  Backup Source:
    Destination:   pvc/xb-cron-ope04-fs-pvc-2024560037-372f8
    Storage Name:  fs-pvc
  Pitr:
    Backup Source:
      s3:
        Bucket:              pitr-backup
        Credentials Secret:  pxc-minio
        Endpoint URL:        http://pxc-minio:9000
    Type:                    latest
  Pxc Cluster:               ope04
Status:
  Comments:  unknown storage type
  State:     Failed
Events:      <none>

Restore attempts, all failed with same error

cat <<EOF | kubectl apply -f-
apiVersion: "pxc.percona.com/v1"
kind: "PerconaXtraDBClusterRestore"
metadata:
  name: "pitrtestlatest4"
spec:
  pxcCluster: ope04
  backupSource:
    destination: pvc/xb-cron-ope04-fs-pvc-2024520037-372f8
    storageName: fs-pvc
  pitr:
    type: latest
    backupSource:
      s3:
        bucket: pitr-backup
        credentialsSecret: pxc-minio
        endpointUrl: http://pxc-minio:9000
EOF

cat <<EOF | kubectl apply -f-
apiVersion: "pxc.percona.com/v1"
kind: "PerconaXtraDBClusterRestore"
metadata:
  name: "pitrtestlatest9"
spec:
  pxcCluster: ope04
  backupSource:
    destination: "pvc/xb-cron-ope04-fs-pvc-2024530037-372f8"
    storageName: "fs-pvc"
  pitr:
    type: latest
    backupSource:
     storageName: "bcp-s3"
EOF

cat <<EOF | kubectl apply -f-
apiVersion: "pxc.percona.com/v1"
kind: "PerconaXtraDBClusterRestore"
metadata:
  name: "pitr-resttest5"
spec:
  pxcCluster: "ope04"
  backupName: "cron-ope04-fs-pvc-2024570052-372f8"
  pitr:
    type: date
    date: "2024-05-07 09:00:00"
    backupSource:
      storageName: bcp-s3
EOF
1 Like