Restore of Postgresql database is not working

Percona Everest
1

Hello,
I tried to search but could not find any helpful information about this issue. When I try to restore Postgresql database from previous backup, I find this message in pod log

pgbackrest restore --set= --type=immediate --stanza=db --pg1-path=/pgdata/pg16 --repo=2 --delta --link-map=pg_wal=/pgdata/pg16_wal
2024-08-02T08:51:49.678993830Z ERROR: [032]: ‘’ must be >= 1 character for ‘set’ option

and after a while status of database is back to up but database is not restored, status of restore is set to “Failed”.

Also when I try to run “Create DB from a backup” from the same backup, new database is created but status is “Unknown” and I cannot notice any further activity, only thing I can do is to delete database.

Do you have an idea where I can look further to resolve these issues?

Thank you and best regards!

2

@sa7411 could you please walk me through on how you get into this situation?
What are the steps to reproduce, which versions do you use (k8s, everest), what are the building blocks (which S3 storage is that)?

3

Hi @sa7411! To investigate the problem it would be really helpful to see the resources configuration:

  1. the backup that causes the problems (everest and pg)
kubectl get dbb <backup_name> -n <your_namespace> -o yaml
kubectl get pg-backup <backup_name> -n <your_namespace> -o yaml
  1. the restore that causes the problems (everest and pg)
kubectl get dbr <restore_name> -n <your_namespace> -o yaml
kubectl get pg-restore <restore_name> -n <your_namespace> -o yaml
  1. DB which is created from the backup and stuck in Unknown (everest and pg)
kubectl get db <db_cluster_name> -n <your_namespace> -o yaml
kubectl get pg <db_cluster_name> -n <your_namespace> -o yaml

Notes:
<your_namespace> - the namespace which is set when installing Everest, by default it’s everest

4

Hello, thank you for your reply,

these are the outputs of commands

1. the backup that causes the problems (everest and pg)
it is any backup, not just this one…

kubectl get dbb postgresql-dss-repo2-full-wrstp -n everest -o yaml

apiVersion: everest.percona.com/v1alpha1
kind: DatabaseClusterBackup
metadata:
creationTimestamp: “2024-08-15T04:00:41Z”
generation: 1
labels:
backupStorage-minio-: used
clusterName: postgresql-dss
name: postgresql-dss-repo2-full-wrstp
namespace: everest
resourceVersion: “46542986”
uid: 6f831706-fa1b-454b-977b-860474ee1f2f
spec:
backupStorageName: minio-
dbClusterName: postgresql-dss
status:
completed: “2024-08-15T04:01:00Z”
created: “2024-08-15T04:00:41Z”
gaps: false
state: Succeeded

kubectl get pg-backup postgresql-dss-repo2-full-wrstp -n everest -o yaml

apiVersion: pgv2.percona.com/v2
kind: PerconaPGBackup
metadata:
creationTimestamp: “2024-08-15T04:00:41Z”
generateName: postgresql-dss-repo2-full-
generation: 1
name: postgresql-dss-repo2-full-wrstp
namespace: everest
ownerReferences:

  • apiVersion: everest.percona.com/v1alpha1
    blockOwnerDeletion: true
    kind: DatabaseClusterBackup
    name: postgresql-dss-repo2-full-wrstp
    uid: 6f831706-fa1b-454b-977b-860474ee1f2f
    resourceVersion: “46542985”
    uid: 4197399d-a501-40ff-9453-5c7c68e13775
    spec:
    options:
  • –type=full
    pgCluster: postgresql-dss
    repoName: repo2
    status:
    backupType: full
    completed: “2024-08-15T04:01:00Z”
    destination: s3://percona/postgresql-dss/c08bc6ca-2c8d-4d7a-b611-56d0cc84fffe
    image: percona/percona-postgresql-operator:2.3.1-ppg16-pgbackrest
    jobName: postgresql-dss-backup-dtng
    repo:
    name: repo2
    s3:
    bucket: percona
    endpoint: minio.:9000/
    region: ba-central-1
    schedules:
    full: 0 4 * * *
    state: Succeeded
    storageType: s3

2. the restore that causes the problems (everest and pg)

kubectl get dbr restore-67w -n everest -o yaml

apiVersion: everest.percona.com/v1alpha1
kind: DatabaseClusterRestore
metadata:
creationTimestamp: “2024-08-15T05:46:21Z”
generation: 1
labels:
clusterName: postgresql-dss
name: restore-67w
namespace: everest
ownerReferences:

  • apiVersion: everest.percona.com/v1alpha1
    blockOwnerDeletion: true
    kind: DatabaseCluster
    name: postgresql-dss
    uid: c08bc6ca-2c8d-4d7a-b611-56d0cc84fffe
    resourceVersion: “46602855”
    uid: fb0b1ce0-cbd4-4e88-af5a-6af82547c5bb
    spec:
    dataSource:
    dbClusterBackupName: postgresql-dss-repo2-full-wrstp
    dbClusterName: postgresql-dss
    status:
    state: Failed

kubectl get pg-restore restore-67w -n everest -o yaml

apiVersion: pgv2.percona.com/v2
kind: PerconaPGRestore
metadata:
creationTimestamp: “2024-08-15T05:46:22Z”
generation: 1
name: restore-67w
namespace: everest
ownerReferences:

  • apiVersion: everest.percona.com/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: DatabaseClusterRestore
    name: restore-67w
    uid: fb0b1ce0-cbd4-4e88-af5a-6af82547c5bb
    resourceVersion: “46602823”
    uid: c58b6897-7ee4-4146-a298-d6876f5fb47c
    spec:
    options:
  • –set=
  • –type=immediate
    pgCluster: postgresql-dss
    repoName: repo2
    status:
    state: Failed

3. DB which is created from the backup and stuck in Unknown (everest and pg)

kubectl get db restored-postgresql-ds -n everest -o yaml

apiVersion: everest.percona.com/v1alpha1
kind: DatabaseCluster
metadata:
creationTimestamp: “2024-08-15T06:01:55Z”
finalizers:

  • everest.percona.com/dbb-cleanup
  • foregroundDeletion
    generation: 3
    labels:
    clusterName: restored-postgresql-ds
    monitoringConfigName: pmm
    name: restored-postgresql-ds
    namespace: everest
    resourceVersion: “46605147”
    uid: d9837be7-ba76-48d4-8329-85b60b9c9ae8
    spec:
    allowUnsafeConfiguration: true
    backup:
    enabled: false
    pitr:
    enabled: false
    dataSource:
    dbClusterBackupName: postgresql-dss-repo2-full-wrstp
    engine:
    replicas: 1
    resources:
    cpu: “1”
    memory: 2G
    storage:
    class: trident
    size: 25G
    type: postgresql
    userSecretsName: everest-secrets-restored-postgresql-ds
    version: “16.1”
    monitoring:
    monitoringConfigName: pmm
    resources: {}
    proxy:
    expose:
    ipSourceRanges:
    • 0.0.0.0/0
      type: external
      replicas: 1
      resources:
      cpu: “0”
      memory: “0”

kubectl get perconapgclusters restored-postgresql-ds -n everest -o yaml

Error from server (NotFound): perconapgclusters.pgv2.percona.com “restored-postgresql-ds” not found

5

Hi @sa7411

Thank you for providing the configurations, it helped a lot.

Everest lists the objects in the storage to figure out the PG backups information. From the provided spec it looks like there are some problems in listing the objects in the storage (probably no permission but there could be other reasons), which causes the dbb.spec.status.destination is empty.

The everest-operator pod logs would help to figure out the exact reason why Everest can’t list the objects in the storage.

This command greps for the logs that are related to the issue:

kubectl logs $(kubectl get po --selector "app.kubernetes.io/name=everest-operator" -n everest-system  -o jsonpath='{.items[*].metadata.name}') -n everest-system | grep -e  "unable to get backup storage secret" -e "unable to load AWS configuration" -e "unable to list objects in bucket" -e "no backup found in bucket"

if nothing is found I would suggest get the full log as a file:

kubectl logs $(kubectl get po --selector "app.kubernetes.io/name=everest-operator" -n everest-system  -o jsonpath='{.items[*].metadata.name}') -n everest-system > everest-operator-logs.txt

and attach the everest-operator-logs.txt so we could analyze it.

Note: Everest 1.2 release is planned to include an upgrade for the PG operator which will allow to get rid of that listing, so the problem should also be gone.

6

Hello, thank you very much for response. This helped me to find the reason for restore failing. It was backup storage endpoint, it was missing “https://” prefix, although backup itself did not report any issues. Maybe it would be good idea to have some kind of validation on UI.

Best Regards!

1 Like
7

For future reference, here’s the JIRA ticket we use for internal tracking.