Description:
It is not possible to do PITR restore when full database backup is on filesystem and pitr pod holding binlogs on S3 minio.
Restore ends with error and “unknown storage type”
Restore normally works if:
- it is not PITR
- it is PITR but both backup and binlogs are on S3 minio storage
Steps to Reproduce:
- configure backup on filesystem
- enable pitr and configure to S3 minio
- do manual backup on filesystem or wait conjob to make one
- do some transactions in the database to verify
- verify the transaction is in binlog on S3 minio
- run PITR restore
Version:
oc version
Client Version: 4.12.28
Kustomize Version: v4.5.7
Server Version: 4.12.28
Kubernetes Version: v1.25.11+1485cc9
+-------------+
| @@version |
+-------------+
| 8.0.32-24.2 |
+-------------+
percona-xtradb-cluster-operator:1.13.0-pxc8.0-backup-pxb8.0.32
Logs:
none
Expected Result:
Database to be restored to latest available transaction in latest available binlog
Actual Result:
The result is incomplete restore, 3 haproxy pods started but only 1 pxc pod.
NAME READY STATUS RESTARTS AGE
ope04-haproxy-0 2/2 Running 0 24h
ope04-haproxy-1 2/2 Running 0 23h
ope04-haproxy-2 2/2 Running 0 23h
ope04-pitr-5dcb96459-gdptq 1/1 Running 0 23h
ope04-pxc-0 3/3 Running 0 23h
ope04-pxc-operator-7b9677c5b9-bsq88 1/1 Running 0 8d
pxc-minio-5555f54887-k2rt6 1/1 Running 0 7d22h
All attempts to scale up remaining two pxc pods leads pods to be in pending state
status:
conditions:
- lastProbeTime: null
lastTransitionTime: "2024-04-24T20:07:46Z"
message: '0/6 nodes are available: 3 node(s) didn''t find available persistent
volumes to bind, 3 node(s) had untolerated taint {node-role.kubernetes.io/master:
}. preemption: 0/6 nodes are available: 6 Preemption is not helpful for scheduling.'
reason: Unschedulable
status: "False"
type: PodScheduled
phase: Pending
qosClass: Burstable
In order to scale up pxc pods, it was necessary to recreate PVs on which those pods are running
Additional Information:
PXC operator backup yaml
pitr:
enabled: true
resources: {}
storageName: bcp-s3
timeBetweenUploads: 60
schedule:
- keep: 3
name: daily-backup
schedule: 0 0 * * *
storageName: fs-pvc
serviceAccountName: percona-xtradb-cluster-operator
storages:
bcp-s3:
podSecurityContext:
supplementalGroups:
- 1001
resources: {}
s3:
bucket: pitr-backup
credentialsSecret: pxc-minio
endpointUrl: http://pxc-minio:9000
type: s3
fs-pvc:
podSecurityContext:
fsGroup: 1001
supplementalGroups:
- 1001
- 1002
- 1003
resources: {}
type: filesystem
volume:
persistentVolumeClaim:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 50Gi
storageClassName: rook-minio
Backups
k get pxc-backup
NAME CLUSTER STORAGE DESTINATION STATUS COMPLETED AGE
cron-ope04-fs-pvc-2024550037-372f8 ope04 fs-pvc pvc/xb-cron-ope04-fs-pvc-2024550037-372f8 Succeeded 2d20h 2d20h
cron-ope04-fs-pvc-2024560037-372f8 ope04 fs-pvc pvc/xb-cron-ope04-fs-pvc-2024560037-372f8 Succeeded 44h 44h
cron-ope04-fs-pvc-2024570052-372f8 ope04 fs-pvc pvc/xb-cron-ope04-fs-pvc-2024570052-372f8 Succeeded 20h 20h
s3testbackup1 ope04 bcp-s3 s3://pitr-backup/ope04-2024-05-06-16:10:53-full Succeeded 28h 28h
Storage
k get pxc-backup
NAME CLUSTER STORAGE DESTINATION STATUS COMPLETED AGE
cron-ope04-fs-pvc-2024550037-372f8 ope04 fs-pvc pvc/xb-cron-ope04-fs-pvc-2024550037-372f8 Succeeded 2d20h 2d20h
cron-ope04-fs-pvc-2024560037-372f8 ope04 fs-pvc pvc/xb-cron-ope04-fs-pvc-2024560037-372f8 Succeeded 44h 44h
cron-ope04-fs-pvc-2024570052-372f8 ope04 fs-pvc pvc/xb-cron-ope04-fs-pvc-2024570052-372f8 Succeeded 20h 20h
s3testbackup1 ope04 bcp-s3 s3://pitr-backup/ope04-2024-05-06-16:10:53-full Succeeded 28h 28h
Restore
Spec:
Backup Source:
Destination: pvc/xb-cron-ope04-fs-pvc-2024560037-372f8
Storage Name: fs-pvc
Pitr:
Backup Source:
s3:
Bucket: pitr-backup
Credentials Secret: pxc-minio
Endpoint URL: http://pxc-minio:9000
Type: latest
Pxc Cluster: ope04
Status:
Comments: unknown storage type
State: Failed
Events: <none>
Restore attempts, all failed with same error
cat <<EOF | kubectl apply -f-
apiVersion: "pxc.percona.com/v1"
kind: "PerconaXtraDBClusterRestore"
metadata:
name: "pitrtestlatest4"
spec:
pxcCluster: ope04
backupSource:
destination: pvc/xb-cron-ope04-fs-pvc-2024520037-372f8
storageName: fs-pvc
pitr:
type: latest
backupSource:
s3:
bucket: pitr-backup
credentialsSecret: pxc-minio
endpointUrl: http://pxc-minio:9000
EOF
cat <<EOF | kubectl apply -f-
apiVersion: "pxc.percona.com/v1"
kind: "PerconaXtraDBClusterRestore"
metadata:
name: "pitrtestlatest9"
spec:
pxcCluster: ope04
backupSource:
destination: "pvc/xb-cron-ope04-fs-pvc-2024530037-372f8"
storageName: "fs-pvc"
pitr:
type: latest
backupSource:
storageName: "bcp-s3"
EOF
cat <<EOF | kubectl apply -f-
apiVersion: "pxc.percona.com/v1"
kind: "PerconaXtraDBClusterRestore"
metadata:
name: "pitr-resttest5"
spec:
pxcCluster: "ope04"
backupName: "cron-ope04-fs-pvc-2024570052-372f8"
pitr:
type: date
date: "2024-05-07 09:00:00"
backupSource:
storageName: bcp-s3
EOF