# PITR restore fails when backup is on FS and binlogs on S3 minio

**URL:** <https://forums.percona.com/t/pitr-restore-fails-when-backup-is-on-fs-and-binlogs-on-s3-minio/30109>\
**Category:** Percona Operator for MySQL\
**Created:** [May 7, 2024, 9:50pm UTC](https://forums.percona.com/t/pitr-restore-fails-when-backup-is-on-fs-and-binlogs-on-s3-minio/30109 "2024-05-07T21:50:00Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Roman\_J](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/roman_j/32/16338_2.png) [@Roman\_J](https://forums.percona.com/u/Roman_J)\
**Post date:** [May 7, 2024, 9:50pm UTC](https://forums.percona.com/t/pitr-restore-fails-when-backup-is-on-fs-and-binlogs-on-s3-minio/30109/1 "2024-05-07T21:50:00Z")

</div>

## Description:

It is not possible to do PITR restore when full database backup is on filesystem and pitr pod holding binlogs on **S3 minio**.  
Restore ends with error and “unknown storage type”

Restore normally works if:

- it is not PITR
- it is PITR but both backup and binlogs are on S3 minio storage

## Steps to Reproduce:

1. configure backup on **filesystem**
2. enable pitr and configure to **S3 minio**
3. do manual backup on filesystem or wait conjob to make one
4. do some transactions in the database to verify
5. verify the transaction is in binlog on S3 minio
6. run PITR restore

## Version:

```auto
oc version
Client Version: 4.12.28
Kustomize Version: v4.5.7
Server Version: 4.12.28
Kubernetes Version: v1.25.11+1485cc9
+-------------+
| @@version |
+-------------+
| 8.0.32-24.2 |
+-------------+
percona-xtradb-cluster-operator:1.13.0-pxc8.0-backup-pxb8.0.32

```

## Logs:

none

## Expected Result:

Database to be restored to latest available transaction in latest available binlog

## Actual Result:

The result is incomplete restore, 3 haproxy pods started but only 1 pxc pod.

```auto
NAME READY STATUS RESTARTS AGE
ope04-haproxy-0 2/2 Running 0 24h
ope04-haproxy-1 2/2 Running 0 23h
ope04-haproxy-2 2/2 Running 0 23h
ope04-pitr-5dcb96459-gdptq 1/1 Running 0 23h
ope04-pxc-0 3/3 Running 0 23h
ope04-pxc-operator-7b9677c5b9-bsq88 1/1 Running 0 8d
pxc-minio-5555f54887-k2rt6 1/1 Running 0 7d22h

```

All attempts to scale up remaining two pxc pods leads pods to be in pending state

```auto
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2024-04-24T20:07:46Z"
    message: '0/6 nodes are available: 3 node(s) didn''t find available persistent
      volumes to bind, 3 node(s) had untolerated taint {node-role.kubernetes.io/master:
      }. preemption: 0/6 nodes are available: 6 Preemption is not helpful for scheduling.'
    reason: Unschedulable
    status: "False"
    type: PodScheduled
  phase: Pending
  qosClass: Burstable

```

In order to scale up pxc pods, it was necessary to recreate PVs on which those pods are running

## Additional Information:

PXC operator backup yaml

```auto
      pitr:
        enabled: true
        resources: {}
        storageName: bcp-s3
        timeBetweenUploads: 60
      schedule:
      - keep: 3
        name: daily-backup
        schedule: 0 0 * * *
        storageName: fs-pvc
      serviceAccountName: percona-xtradb-cluster-operator
      storages:
        bcp-s3:
          podSecurityContext:
            supplementalGroups:
            - 1001
          resources: {}
          s3:
            bucket: pitr-backup
            credentialsSecret: pxc-minio
            endpointUrl: http://pxc-minio:9000
          type: s3
        fs-pvc:
          podSecurityContext:
            fsGroup: 1001
            supplementalGroups:
            - 1001
            - 1002
            - 1003
          resources: {}
          type: filesystem
          volume:
            persistentVolumeClaim:
              accessModes:
              - ReadWriteOnce
              resources:
                requests:
                  storage: 50Gi
              storageClassName: rook-minio

```

Backups

```auto
k get pxc-backup
NAME CLUSTER STORAGE DESTINATION STATUS COMPLETED AGE
cron-ope04-fs-pvc-2024550037-372f8 ope04 fs-pvc pvc/xb-cron-ope04-fs-pvc-2024550037-372f8 Succeeded 2d20h 2d20h
cron-ope04-fs-pvc-2024560037-372f8 ope04 fs-pvc pvc/xb-cron-ope04-fs-pvc-2024560037-372f8 Succeeded 44h 44h
cron-ope04-fs-pvc-2024570052-372f8 ope04 fs-pvc pvc/xb-cron-ope04-fs-pvc-2024570052-372f8 Succeeded 20h 20h
s3testbackup1 ope04 bcp-s3 s3://pitr-backup/ope04-2024-05-06-16:10:53-full Succeeded 28h 28h

```

Storage

```auto
k get pxc-backup
NAME CLUSTER STORAGE DESTINATION STATUS COMPLETED AGE
cron-ope04-fs-pvc-2024550037-372f8 ope04 fs-pvc pvc/xb-cron-ope04-fs-pvc-2024550037-372f8 Succeeded 2d20h 2d20h
cron-ope04-fs-pvc-2024560037-372f8 ope04 fs-pvc pvc/xb-cron-ope04-fs-pvc-2024560037-372f8 Succeeded 44h 44h
cron-ope04-fs-pvc-2024570052-372f8 ope04 fs-pvc pvc/xb-cron-ope04-fs-pvc-2024570052-372f8 Succeeded 20h 20h
s3testbackup1 ope04 bcp-s3 s3://pitr-backup/ope04-2024-05-06-16:10:53-full Succeeded 28h 28h

```

Restore

```auto
Spec:
  Backup Source:
    Destination: pvc/xb-cron-ope04-fs-pvc-2024560037-372f8
    Storage Name: fs-pvc
  Pitr:
    Backup Source:
      s3:
        Bucket: pitr-backup
        Credentials Secret: pxc-minio
        Endpoint URL: http://pxc-minio:9000
    Type: latest
  Pxc Cluster: ope04
Status:
  Comments: unknown storage type
  State: Failed
Events: <none>

```

Restore attempts, all failed with same error

```auto
cat <<EOF | kubectl apply -f-
apiVersion: "pxc.percona.com/v1"
kind: "PerconaXtraDBClusterRestore"
metadata:
  name: "pitrtestlatest4"
spec:
  pxcCluster: ope04
  backupSource:
    destination: pvc/xb-cron-ope04-fs-pvc-2024520037-372f8
    storageName: fs-pvc
  pitr:
    type: latest
    backupSource:
      s3:
        bucket: pitr-backup
        credentialsSecret: pxc-minio
        endpointUrl: http://pxc-minio:9000
EOF

cat <<EOF | kubectl apply -f-
apiVersion: "pxc.percona.com/v1"
kind: "PerconaXtraDBClusterRestore"
metadata:
  name: "pitrtestlatest9"
spec:
  pxcCluster: ope04
  backupSource:
    destination: "pvc/xb-cron-ope04-fs-pvc-2024530037-372f8"
    storageName: "fs-pvc"
  pitr:
    type: latest
    backupSource:
     storageName: "bcp-s3"
EOF

cat <<EOF | kubectl apply -f-
apiVersion: "pxc.percona.com/v1"
kind: "PerconaXtraDBClusterRestore"
metadata:
  name: "pitr-resttest5"
spec:
  pxcCluster: "ope04"
  backupName: "cron-ope04-fs-pvc-2024570052-372f8"
  pitr:
    type: date
    date: "2024-05-07 09:00:00"
    backupSource:
      storageName: bcp-s3
EOF

```

---

<div class="post-metadata">

**Author:** ![Slava\_Sarzhan](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/slava_sarzhan/32/3196_2.png) [@Slava\_Sarzhan](https://forums.percona.com/u/Slava_Sarzhan)\
**Post date:** [June 24, 2024, 7:47am UTC](https://forums.percona.com/t/pitr-restore-fails-when-backup-is-on-fs-and-binlogs-on-s3-minio/30109/2 "2024-06-24T07:47:14Z")

</div>

Hi @Roman_J please have a look at our official documentation [Store binary logs for point-in-time recovery - Percona Operator for MySQL](https://docs.percona.com/percona-operator-for-mysql/pxc/backups-pitr.html#store-binary-logs-for-point-in-time-recovery)

`Both binlog and full backup should use s3-compatible storage to make point-in-time recovery work!`

PITR works only in case both binlog and full backup should use s3-compatible storage.