Issues with configuring backup against S3 storage

92fnko · November 12, 2024, 8:37pm

Description:

We have trouble configuring backup against a local MinIO instance. When creating the cluster, the cluster is successfully being created, but the operator reports ERROR unable to create stanza and when trying to run a on-demand backup, nothing happens and the status of backup resource is Starting.

Currently running OpenShift 4.15.

Steps to Reproduce:

Create cluster resource with the pgbackrest parameters against local MinIO instance.
Create a pg-backup resource to trigger on-demand backup.
Status for pg-backup resource is Starting forever.

Version:

v.2.4.0

Logs:

Log from operator:

2024-11-12T17:23:01.313Z        INFO    Waiting for backup to start     {"controller": "perconapgbackup", "controllerGroup": "pgv2.percona.com", "controllerKind": "PerconaPGBackup", "PerconaPGBackup": {"name":"<REDACTED>","namespace":"<REDACTED>"}, "namespace": "<REDACTED>", "name": "<REDACTED>", "reconcileID": "436f40f4-34f1-4dff-b633-08f8fe31780a", "request": {"name":"<REDACTED>","namespace":"<REDACTED>"}}
2024-11-12T17:23:01.648Z        ERROR   get latest backup       {"controller": "perconapgcluster", "controllerGroup": "pgv2.percona.com", "controllerKind": "PerconaPGCluster", "PerconaPGCluster": {"name":"<REDACTED>","namespace":"<REDACTED>"}, "namespace": "<REDACTED>", "name": "<REDACTED>", "reconcileID": "16ba3814-edfd-4046-a67d-df35ae4fbbd7", "error": "no completed backups found", "errorVerbose": "no completed backups found\ngithub.com/percona/percona-postgresql-operator/percona/watcher.getLatestBackup\n\t/go/src/github.com/percona/percona-postgresql-operator/percona/watcher/wal.go:129\ngithub.com/percona/percona-postgresql-operator/percona/watcher.WatchCommitTimestamps\n\t/go/src/github.com/percona/percona-postgresql-operator/percona/watcher/wal.go:65\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1695"}
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1695
2024-11-12T17:23:04.803Z        ERROR   unable to create stanza {"controller": "postgrescluster", "controllerGroup": "postgres-operator.crunchydata.com", "controllerKind": "PostgresCluster", "PostgresCluster": {"name":"<REDACTED>","namespace":"<REDACTED>"}, "namespace": "<REDACTED>", "name": "<REDACTED>", "reconcileID": "3264a2fa-dd1a-4272-b79a-cd9167f45de4", "reconciler": "pgBackRest", "error": "command terminated with exit code 32: ", "errorVerbose": "command terminated with exit code 32: \ngithub.com/percona/percona-postgresql-operator/internal/pgbackrest.Executor.StanzaCreateOrUpgrade\n\t/go/src/github.com/percona/percona-postgresql-operator/internal/pgbackrest/pgbackrest.go:96\ngithub.com/percona/percona-postgresql-operator/internal/controller/postgrescluster.(*Reconciler).reconcileStanzaCreate\n\t/go/src/github.com/percona/percona-postgresql-operator/internal/controller/postgrescluster/pgbackrest.go:2705\ngithub.com/percona/percona-postgresql-operator/internal/controller/postgrescluster.(*Reconciler).reconcilePGBackRest\n\t/go/src/github.com/percona/percona-postgresql-operator/internal/controller/postgrescluster/pgbackrest.go:1412\ngithub.com/percona/percona-postgresql-operator/internal/controller/postgrescluster.(*Reconciler).Reconcile\n\t/go/src/github.com/percona/percona-postgresql-operator/internal/controller/postgrescluster/controller.go:390\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:222\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1695\ngithub.com/percona/percona-postgresql-operator/internal/controller/postgrescluster.(*Reconciler).reconcileStanzaCreate\n\t/go/src/github.com/percona/percona-postgresql-operator/internal/controller/postgrescluster/pgbackrest.go:2712\ngithub.com/percona/percona-postgresql-operator/internal/controller/postgrescluster.(*Reconciler).reconcilePGBackRest\n\t/go/src/github.com/percona/percona-postgresql-operator/internal/controller/postgrescluster/pgbackrest.go:1412\ngithub.com/percona/percona-postgresql-operator/internal/controller/postgrescluster.(*Reconciler).Reconcile\n\t/go/src/github.com/percona/percona-postgresql-operator/internal/controller/postgrescluster/controller.go:390\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:222\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1695"}
github.com/percona/percona-postgresql-operator/internal/controller/postgrescluster.(*Reconciler).Reconcile
        /go/src/github.com/percona/percona-postgresql-operator/internal/controller/postgrescluster/controller.go:390
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:114
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:311
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:261
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:222

Expected Result:

That the backup would succeed.

Actual Result:

The status of the pg-backup resource is stuck on Starting.

Additional Information:

Cluster resource:

...
      configuration:
        - secret:
            name: <REDACTED>-pgbackrest-secrets
      global:
        repo1-retention-full: "14"
        repo1-retention-full-type: time
        repo1-retention-diff: "14"
        repo1-retention-diff-type: time
      manual:
        repoName: repo1
        options:
         - --type=full
      repos:
      - name: repo1
        s3:
          endpoint: "http://<REDACTED>:9000" # yes, our internal MinIO instance is for some reason http only..
          bucket: "<REDACTED>"
          region: minio
...

-pgbackrest-config config map:

...
data:
  pgbackrest-server.conf: ""
  pgbackrest_instance.conf: |
    # Generated by postgres-operator. DO NOT EDIT.
    # Your changes will not be saved.

    [global]
    log-path = /pgdata/pgbackrest/log
    repo1-path = /pgbackrest/repo1
    repo1-retention-diff = 14
    repo1-retention-diff-type = time
    repo1-retention-full = 14
    repo1-retention-full-type = time
    repo1-s3-bucket = <REDACTED>
    repo1-s3-endpoint = http://<REDACTED>:9000
    repo1-s3-region = minio
    repo1-type = s3

    [db]
    pg1-path = /pgdata/pg16
    pg1-port = 5432
    pg1-socket-path = /tmp/postgres
...

mboncalo · November 13, 2024, 9:18am

Did you create your -pgbackrest-secrets secret with the other required S3 credentials ? repo1-s3-key, repo1-s3-key-secret
And other configuration like repo1-s3-uri-style=path, repo1-storage-verify-tls=n

92fnko · November 13, 2024, 1:20pm

Yep!

I have also tried the repo1-s3-uri-style=path (to be honest, I don’t know what this is actually doing) and repo1-storage-verify-tls=n (shouldn’t be needed at all as we using HTTP at the moment).

Topic		Replies	Views
Backups to repo1 not working Percona Operator for PostgreSQL	2	39	September 8, 2024
pgBackRest and s3 Panroni and operator Percona Operator for PostgreSQL	3	1423	August 8, 2022
V2.2 not work pgbackrest Percona Operator for PostgreSQL	3	1008	August 8, 2023
Can't get backup working (operator crashes) Percona Operator for MongoDB closed-no-reply	0	750	September 1, 2021
Backup Not starting Percona Operator for MySQL	5	811	August 9, 2021