Has anyone been able to get S3 backups working for Postgres Operator 1.3?

I followed these instructions and no matter what I seem to do, I get the following errors:

Error Logs:
time=“2023-03-25T17:57:03Z” level=info msg=“crunchy-pgbackrest starts”
time=“2023-03-25T17:57:03Z” level=info msg=“debug flag set to false”
time=“2023-03-25T17:57:03Z” level=info msg=“backrest backup command requested”
time=“2023-03-25T17:57:03Z” level=info msg=“backrest command will be executed for both local and s3 storage”
time=“2023-03-25T17:57:03Z” level=info msg=“command to execute is [pgbackrest backup --stanza=db --type=full --repo1-retention-full=2 --db-host=10.128.2.56 --db-path=/pgdata/dev-eng-pg-cluster-repl3 ; pgbackrest backup --stanza=db --type=full --repo1-retention-full=2 --db-host=10.128.2.56 --db-path=/pgdata/dev-eng-pg-cluster-repl3 --repo1-type=s3 --no-repo1-s3-verify-tls]”
time=“2023-03-25T17:57:17Z” level=info msg=“output=”
time=“2023-03-25T17:57:17Z” level=info msg=“stderr=[ERROR: [037]: backup command requires option: repo1-s3-bucket\n]”
time=“2023-03-25T17:57:17Z” level=fatal msg=“command terminated with exit code 37”

Thanks in advance!

Hi

I believe the issue submitted correlates with https://forums.percona.com/t/help-configuring-postgres-operator-backup-section-of-cr-yaml . So please, share your CR backup section and make sure that there is no s3:// in backup settings.

Thanks for the reply. I have tried using different forms of the bucket name with no luck, hopefully you see something I am missing. Below is my latest attempt at the cr section:

    storages:
      s3-us-west:
        type: s3
        bucket: my-backup-bucket
        region: us-west-2
        endpointUrl: s3.amazonaws.com
        verifyTLS: true
        uriStyle: path

   schedule:
      - name: "s3-backup"
        schedule: "*/3 * * * *"
        keep: 2
        type: full
        storage: s3-us-west

Contents of schedule configmap that is created:
{“version”:“”,“name”:“s3-backup”,“created”:“0001-01-01T00:00:00Z”,“schedule”:“*/3 * * * *”,“namespace”:“pgo”,“type”:“pgbackrest”,“cluster”:“dev-eng-pg-cluster”,“pgbackrest”:{“deployment”:“dev-eng-pg-cluster”,“label”:“”,“container”:“database”,“type”:“full”,“storageType”:“s3”,“options”:“–repo1-retention-full=2”},“policy”:{“secret”:“”,“name”:“”,“imagePrefix”:“”,“imageTag”:“”,“database”:“”}}

Error log:
Wed Apr 5 13:42:01 UTC 2023 INFO: Image mode found: pgbackrest
Wed Apr 5 13:42:01 UTC 2023 INFO: Starting in ‘pgbackrest’ mode
time=“2023-04-05T13:42:01Z” level=info msg=“crunchy-pgbackrest starts”
time=“2023-04-05T13:42:01Z” level=info msg=“debug flag set to false”
time=“2023-04-05T13:42:01Z” level=info msg=“backrest backup command requested”
time=“2023-04-05T13:42:01Z” level=info msg=“backrest command will be executed for both local and s3 storage”
time=“2023-04-05T13:42:01Z” level=info msg=“command to execute is [pgbackrest backup --stanza=db --type=full --repo1-retention-full=2 --db-host=10.129.2.13 --db-path=/pgdata/dev-eng-pg-cluster-repl2 ; pgbackrest backup --stanza=db --type=full --repo1-retention-full=2 --db-host=10.129.2.13 --db-path=/pgdata/dev-eng-pg-cluster-repl2 --repo1-type=s3 --no-repo1-s3-verify-tls]”
time=“2023-04-05T13:42:14Z” level=info msg=“output=
time=“2023-04-05T13:42:14Z” level=info msg=“stderr=[ERROR: [037]: backup command requires option: repo1-s3-bucket\n]”
time=“2023-04-05T13:42:14Z” level=fatal msg=“command terminated with exit code 37”

Michael,

Did you try to recreate the cluster? It’s important since the pgbackrests’ overall configuration in 1.3.0 does not allow the user to change the vast majority of backup/restore related settings.

Interesting, so if we need to change storages the cluster needs to be recreated? I am assuming schedules can be changed at anytime? That is good to know. Before I do that, where would an s3 bucket prefix go in the configuration? I didn’t see a prefix field as an option so can it be part of the bucket name field, ie. my-bucket-name/folder1/folder2?

Thanks for the help…

Well, it’s the major design limitation of 1.x pg operator family which is a reason why pg operator 2.0.0 (beta) came out. 2.0.0 allows to change backup settings.
If you need to offset the root of inside your bucket feel free to use repoPath option. It will allow you to add intermediate folders to bucket name.

Thanks again for the reply and it looks like we are getting closer. I specified the repoPath /dev/postgres_cluster and during cluster creation it does look like the s3 bucket is getting updated with information. But eventually the backuprest pod errors out with the below error. In AWS both the files indicated below as missing do actually exist. Any thoughts?

Wed Apr 5 16:45:29 UTC 2023 INFO: Image mode found: pgbackrest
Wed Apr 5 16:45:29 UTC 2023 INFO: Starting in ‘pgbackrest’ mode
time=“2023-04-05T16:45:30Z” level=info msg=“crunchy-pgbackrest starts”
time=“2023-04-05T16:45:30Z” level=info msg=“debug flag set to false”
time=“2023-04-05T16:45:30Z” level=info msg=“backrest backup command requested”
time=“2023-04-05T16:45:30Z” level=info msg=“backrest command will be executed for both local and s3 storage”
time=“2023-04-05T16:45:30Z” level=info msg=“command to execute is [pgbackrest backup --type=full --db-host=10.129.2.152 --db-path=/pgdata/dev-pg-cluster ; pgbackrest backup --type=full --db-host=10.129.2.152 --db-path=/pgdata/dev-pg-cluster --repo1-type=s3]”
time=“2023-04-05T16:46:38Z” level=info msg=“output=
time=“2023-04-05T16:46:38Z” level=info msg="stderr=[WARN: option ‘repo1-retention-full’ is not set for ‘repo1-retention-full-type=count’, the repository may run out of space\n HINT: to retain full backups indefinitely (without warning), set option ‘repo1-retention-full’ to the maximum.\nERROR: [055]: unable to load info file ‘/dev/postgres-cluster/backup/db/backup.info’ or ‘/dev/postgres-cluster/backup/db/backup.info.copy’:\n FileMissingError: unable to open missing file ‘/dev/postgres-cluster/backup/db/backup.info’ for read\n FileMissingError: unable to open missing file ‘/dev/postgres-cluster/backup/db/backup.info.copy’ for read\n HINT: backup.info cannot be opened and is required to perform a backup.\n HINT: has a stanza-create been performed?\nWARN: option ‘repo1-retention-full’ is not set for ‘repo1-retention-full-type=count’, the repository may run out of space\n HINT: to retain full backups indefinitely (without warning), set option 'repo1-retention-…
time=“2023-04-05T16:46:38Z” level=fatal msg=“command terminated with exit code 82”

Michael

Has stanza pod finished successfully with your repoPath option value?
Basically it initiates all backup storages on cluster startup. Your change to the backup configuration should be processed by stanza, however to make it happen you have to recreate the cluster.

Yes, the stanza created just fine and it looks like it sent a backup to AWS but then that error.

I decided to move to version 1.4 of the operator to see if this helps but now I am getting permission issues as I am installing on OpenShift.

2023-04-05 18:47:55,658 INFO: No PostgreSQL configuration items changed, nothing to reload.
2023-04-05 18:47:55,663 INFO: Lock owner: None; I am dev-eng-pg-cluster-99665545b-qvv65
2023-04-05 18:47:55,727 INFO: trying to bootstrap a new cluster
The files belonging to this database system will be owned by user “postgres”.
This user must also own the server process.
The database cluster will be initialized with locale “en_US.utf-8”.
The default text search configuration will be set to “english”.
Data page checksums are enabled.
creating directory /pgdata/dev-eng-pg-cluster … initdb: error: could not create directory “/pgdata/dev-eng-pg-cluster”: Permission denied
pg_ctl: database system initialization failed
2023-04-05 18:47:55,796 INFO: removing initialize key after failed attempt to bootstrap the cluster

Please post yaml manifest of the failed pod here.
We need to check it’s security settings.

Here you go, thanks.

kind: Pod
apiVersion: v1
metadata:
  generateName: dev-eng-pg-cluster-99665545b-
  annotations:
    k8s.ovn.org/pod-networks: >-
      {"default":{"ip_addresses":["10.129.2.169/23"],"mac_address":"0a:58:0a:81:02:a9","gateway_ips":["10.129.2.1"],"ip_address":"10.129.2.169/23","gateway_ip":"10.129.2.1"}}
    k8s.v1.cni.cncf.io/network-status: |-
      [{
          "name": "ovn-kubernetes",
          "interface": "eth0",
          "ips": [
              "10.129.2.169"
          ],
          "mac": "0a:58:0a:81:02:a9",
          "default": true,
          "dns": {}
      }]
    k8s.v1.cni.cncf.io/networks-status: |-
      [{
          "name": "ovn-kubernetes",
          "interface": "eth0",
          "ips": [
              "10.129.2.169"
          ],
          "mac": "0a:58:0a:81:02:a9",
          "default": true,
          "dns": {}
      }]
    keep-backups: 'true'
    keep-data: 'true'
    openshift.io/scc: anyuid
    status: >-
      {"conn_url":"postgres://10.129.2.169:5432/postgres","api_url":"http://10.129.2.169:8009/patroni","state":"stopped","role":"uninitialized","version":"2.1.4"}
  resourceVersion: '32456817'
  name: dev-eng-pg-cluster-99665545b-6txbx
  uid: f377f440-0f7e-45e3-ad39-854b7ac96f15
  creationTimestamp: '2023-04-05T19:08:58Z'
  managedFields:
    - manager: Go-http-client
      operation: Update
      apiVersion: v1
      time: '2023-04-05T19:08:58Z'
      fieldsType: FieldsV1
      fieldsV1:
        'f:metadata':
          'f:labels':
            'f:service-name': {}
    - manager: kube-controller-manager
      operation: Update
      apiVersion: v1
      time: '2023-04-05T19:08:58Z'
      fieldsType: FieldsV1
      fieldsV1:
        'f:metadata':
          'f:annotations':
            .: {}
            'f:keep-backups': {}
            'f:keep-data': {}
          'f:generateName': {}
          'f:labels':
            'f:pod-template-hash': {}
            'f:pgo-pg-database': {}
            'f:pg-pod-anti-affinity': {}
            'f:crunchy-pgha-scope': {}
            .: {}
            'f:pgo-version': {}
            'f:pgouser': {}
            'f:pg-cluster': {}
            'f:vendor': {}
            'f:name': {}
            'f:deployment-name': {}
          'f:ownerReferences':
            .: {}
            'k:{"uid":"1dcd828f-1630-46af-a3b6-611580ffb292"}': {}
        'f:spec':
          'f:volumes':
            'k:{"name":"sshd"}':
              .: {}
              'f:name': {}
              'f:secret':
                .: {}
                'f:defaultMode': {}
                'f:secretName': {}
            'k:{"name":"pgdata"}':
              .: {}
              'f:name': {}
              'f:persistentVolumeClaim':
                .: {}
                'f:claimName': {}
            'k:{"name":"podinfo"}':
              .: {}
              'f:downwardAPI':
                .: {}
                'f:defaultMode': {}
                'f:items': {}
              'f:name': {}
            'k:{"name":"primary-volume"}':
              .: {}
              'f:name': {}
              'f:secret':
                .: {}
                'f:defaultMode': {}
                'f:secretName': {}
            'k:{"name":"tmp"}':
              .: {}
              'f:emptyDir':
                .: {}
                'f:medium': {}
                'f:sizeLimit': {}
              'f:name': {}
            'k:{"name":"ssh-config"}':
              .: {}
              'f:name': {}
              'f:secret':
                .: {}
                'f:defaultMode': {}
                'f:items': {}
                'f:secretName': {}
            'k:{"name":"tls-replication"}':
              .: {}
              'f:emptyDir':
                .: {}
                'f:medium': {}
                'f:sizeLimit': {}
              'f:name': {}
            .: {}
            'k:{"name":"user-volume"}':
              .: {}
              'f:name': {}
              'f:secret':
                .: {}
                'f:defaultMode': {}
                'f:secretName': {}
            'k:{"name":"pgbackrest-config"}':
              .: {}
              'f:name': {}
              'f:projected':
                .: {}
                'f:defaultMode': {}
                'f:sources': {}
            'k:{"name":"root-volume"}':
              .: {}
              'f:name': {}
              'f:secret':
                .: {}
                'f:defaultMode': {}
                'f:secretName': {}
            'k:{"name":"dshm"}':
              .: {}
              'f:emptyDir':
                .: {}
                'f:medium': {}
              'f:name': {}
            'k:{"name":"report"}':
              .: {}
              'f:emptyDir':
                .: {}
                'f:medium': {}
                'f:sizeLimit': {}
              'f:name': {}
            'k:{"name":"tls-server"}':
              .: {}
              'f:name': {}
              'f:projected':
                .: {}
                'f:defaultMode': {}
                'f:sources': {}
            'k:{"name":"pgconf-volume"}':
              .: {}
              'f:name': {}
              'f:projected':
                .: {}
                'f:defaultMode': {}
                'f:sources': {}
          'f:containers':
            'k:{"name":"database"}':
              'f:image': {}
              'f:volumeMounts':
                'k:{"mountPath":"/pgconf"}':
                  .: {}
                  'f:mountPath': {}
                  'f:name': {}
                'k:{"mountPath":"/pgconf/pguser"}':
                  .: {}
                  'f:mountPath': {}
                  'f:name': {}
                'k:{"mountPath":"/sshd"}':
                  .: {}
                  'f:mountPath': {}
                  'f:name': {}
                  'f:readOnly': {}
                'k:{"mountPath":"/tmp"}':
                  .: {}
                  'f:mountPath': {}
                  'f:name': {}
                'k:{"mountPath":"/etc/podinfo"}':
                  .: {}
                  'f:mountPath': {}
                  'f:name': {}
                'k:{"mountPath":"/pgdata"}':
                  .: {}
                  'f:mountPath': {}
                  'f:name': {}
                'k:{"mountPath":"/etc/pgbackrest/conf.d"}':
                  .: {}
                  'f:mountPath': {}
                  'f:name': {}
                'k:{"mountPath":"/dev/shm"}':
                  .: {}
                  'f:mountPath': {}
                  'f:name': {}
                .: {}
                'k:{"mountPath":"/pgconf/tls-replication"}':
                  .: {}
                  'f:mountPath': {}
                  'f:name': {}
                'k:{"mountPath":"/pgconf/tls"}':
                  .: {}
                  'f:mountPath': {}
                  'f:name': {}
                'k:{"mountPath":"/pgconf/pgreplicator"}':
                  .: {}
                  'f:mountPath': {}
                  'f:name': {}
                'k:{"mountPath":"/pgconf/pgsuper"}':
                  .: {}
                  'f:mountPath': {}
                  'f:name': {}
                'k:{"mountPath":"/etc/ssh"}':
                  .: {}
                  'f:mountPath': {}
                  'f:name': {}
                  'f:readOnly': {}
              'f:terminationMessagePolicy': {}
              .: {}
              'f:resources':
                .: {}
                'f:limits':
                  .: {}
                  'f:cpu': {}
                  'f:memory': {}
                'f:requests':
                  .: {}
                  'f:cpu': {}
                  'f:memory': {}
              'f:livenessProbe':
                .: {}
                'f:exec':
                  .: {}
                  'f:command': {}
                'f:failureThreshold': {}
                'f:initialDelaySeconds': {}
                'f:periodSeconds': {}
                'f:successThreshold': {}
                'f:timeoutSeconds': {}
              'f:env':
                'k:{"name":"PGBACKREST_REPO1_S3_BUCKET"}':
                  .: {}
                  'f:name': {}
                  'f:value': {}
                'k:{"name":"PGBACKREST_REPO1_HOST_CMD"}':
                  .: {}
                  'f:name': {}
                  'f:value': {}
                'k:{"name":"PATRONI_SCOPE"}':
                  .: {}
                  'f:name': {}
                  'f:valueFrom':
                    .: {}
                    'f:fieldRef': {}
                'k:{"name":"PGHA_STANDBY"}':
                  .: {}
                  'f:name': {}
                  'f:value': {}
                'k:{"name":"PGBACKREST_PG1_SOCKET_PATH"}':
                  .: {}
                  'f:name': {}
                  'f:value': {}
                'k:{"name":"PGHOST"}':
                  .: {}
                  'f:name': {}
                  'f:value': {}
                'k:{"name":"PGBACKREST_REPO1_PATH"}':
                  .: {}
                  'f:name': {}
                  'f:value': {}
                'k:{"name":"PGBACKREST_PG1_PORT"}':
                  .: {}
                  'f:name': {}
                  'f:value': {}
                'k:{"name":"PATRONI_KUBERNETES_LABELS"}':
                  .: {}
                  'f:name': {}
                  'f:value': {}
                'k:{"name":"PGBACKREST_REPO1_S3_KEY_SECRET"}':
                  .: {}
                  'f:name': {}
                  'f:valueFrom':
                    .: {}
                    'f:secretKeyRef': {}
                'k:{"name":"LD_PRELOAD"}':
                  .: {}
                  'f:name': {}
                  'f:value': {}
                'k:{"name":"MODE"}':
                  .: {}
                  'f:name': {}
                  'f:value': {}
                'k:{"name":"PGBACKREST_REPO1_HOST"}':
                  .: {}
                  'f:name': {}
                  'f:value': {}
                'k:{"name":"PGHA_REPLICA_REINIT_ON_START_FAIL"}':
                  .: {}
                  'f:name': {}
                  'f:value': {}
                'k:{"name":"PGHA_PGBACKREST_S3_VERIFY_TLS"}':
                  .: {}
                  'f:name': {}
                  'f:value': {}
                'k:{"name":"PGHA_SYNC_REPLICATION"}':
                  .: {}
                  'f:name': {}
                  'f:value': {}
                'k:{"name":"PGBACKREST_REPO1_S3_REGION"}':
                  .: {}
                  'f:name': {}
                  'f:value': {}
                'k:{"name":"PATRONI_KUBERNETES_NAMESPACE"}':
                  .: {}
                  'f:name': {}
                  'f:valueFrom':
                    .: {}
                    'f:fieldRef': {}
                'k:{"name":"ENABLE_SSHD"}':
                  .: {}
                  'f:name': {}
                  'f:value': {}
                'k:{"name":"PGBACKREST_REPO1_S3_ENDPOINT"}':
                  .: {}
                  'f:name': {}
                  'f:value': {}
                'k:{"name":"PGHA_USER"}':
                  .: {}
                  'f:name': {}
                  'f:value': {}
                .: {}
                'k:{"name":"BACKREST_SKIP_CREATE_STANZA"}':
                  .: {}
                  'f:name': {}
                  'f:value': {}
                'k:{"name":"PGHA_INIT"}':
                  .: {}
                  'f:name': {}
                  'f:valueFrom':
                    .: {}
                    'f:configMapKeyRef': {}
                'k:{"name":"PGHA_TLS_ONLY"}':
                  .: {}
                  'f:name': {}
                  'f:value': {}
                'k:{"name":"PATRONI_POSTGRESQL_DATA_DIR"}':
                  .: {}
                  'f:name': {}
                  'f:value': {}
                'k:{"name":"PGHA_PGBACKREST_LOCAL_S3_STORAGE"}':
                  .: {}
                  'f:name': {}
                  'f:value': {}
                'k:{"name":"PGHA_TLS_ENABLED"}':
                  .: {}
                  'f:name': {}
                  'f:value': {}
                'k:{"name":"PGHA_PGBACKREST_LOCAL_GCS_STORAGE"}':
                  .: {}
                  'f:name': {}
                  'f:value': {}
                'k:{"name":"PGHA_PGBACKREST"}':
                  .: {}
                  'f:name': {}
                  'f:value': {}
                'k:{"name":"PGHA_PASSWORD_TYPE"}':
                  .: {}
                  'f:name': {}
                'k:{"name":"PGBACKREST_STANZA"}':
                  .: {}
                  'f:name': {}
                  'f:value': {}
                'k:{"name":"NSS_WRAPPER_GROUP"}':
                  .: {}
                  'f:name': {}
                  'f:value': {}
                'k:{"name":"PATRONI_LOG_LEVEL"}':
                  .: {}
                  'f:name': {}
                  'f:value': {}
                'k:{"name":"PATRONI_KUBERNETES_SCOPE_LABEL"}':
                  .: {}
                  'f:name': {}
                  'f:value': {}
                'k:{"name":"PGBACKREST_REPO1_TYPE"}':
                  .: {}
                  'f:name': {}
                  'f:value': {}
                'k:{"name":"PGHA_DATABASE"}':
                  .: {}
                  'f:name': {}
                  'f:value': {}
                'k:{"name":"PGBACKREST_REPO1_S3_CA_FILE"}':
                  .: {}
                  'f:name': {}
                  'f:value': {}
                'k:{"name":"PGBACKREST_DB_PATH"}':
                  .: {}
                  'f:name': {}
                  'f:value': {}
                'k:{"name":"PGBACKREST_REPO1_S3_URI_STYLE"}':
                  .: {}
                  'f:name': {}
                  'f:value': {}
                'k:{"name":"PGHA_PG_PORT"}':
                  .: {}
                  'f:name': {}
                  'f:value': {}
                'k:{"name":"PGBACKREST_REPO1_S3_KEY"}':
                  .: {}
                  'f:name': {}
                  'f:valueFrom':
                    .: {}
                    'f:secretKeyRef': {}
                'k:{"name":"NSS_WRAPPER_PASSWD"}':
                  .: {}
                  'f:name': {}
                  'f:value': {}
                'k:{"name":"PGBACKREST_LOG_PATH"}':
                  .: {}
                  'f:name': {}
                  'f:value': {}
              'f:readinessProbe':
                .: {}
                'f:exec':
                  .: {}
                  'f:command': {}
                'f:failureThreshold': {}
                'f:initialDelaySeconds': {}
                'f:periodSeconds': {}
                'f:successThreshold': {}
                'f:timeoutSeconds': {}
              'f:securityContext':
                .: {}
                'f:allowPrivilegeEscalation': {}
                'f:privileged': {}
                'f:readOnlyRootFilesystem': {}
              'f:terminationMessagePath': {}
              'f:imagePullPolicy': {}
              'f:ports':
                .: {}
                'k:{"containerPort":5432,"protocol":"TCP"}':
                  .: {}
                  'f:containerPort': {}
                  'f:name': {}
                  'f:protocol': {}
                'k:{"containerPort":8009,"protocol":"TCP"}':
                  .: {}
                  'f:containerPort': {}
                  'f:name': {}
                  'f:protocol': {}
              'f:name': {}
          'f:dnsPolicy': {}
          'f:serviceAccount': {}
          'f:restartPolicy': {}
          'f:schedulerName': {}
          'f:terminationGracePeriodSeconds': {}
          'f:serviceAccountName': {}
          'f:enableServiceLinks': {}
          'f:securityContext':
            .: {}
            'f:supplementalGroups': {}
          'f:affinity':
            .: {}
            'f:podAntiAffinity':
              .: {}
              'f:preferredDuringSchedulingIgnoredDuringExecution': {}
    - manager: dev-eng-qccd4-master-1.novalocal
      operation: Update
      apiVersion: v1
      time: '2023-04-05T19:08:59Z'
      fieldsType: FieldsV1
      fieldsV1:
        'f:metadata':
          'f:annotations':
            'f:k8s.ovn.org/pod-networks': {}
    - manager: multus
      operation: Update
      apiVersion: v1
      time: '2023-04-05T19:09:12Z'
      fieldsType: FieldsV1
      fieldsV1:
        'f:metadata':
          'f:annotations':
            'f:k8s.v1.cni.cncf.io/network-status': {}
            'f:k8s.v1.cni.cncf.io/networks-status': {}
      subresource: status
    - manager: Patroni
      operation: Update
      apiVersion: v1
      time: '2023-04-05T19:09:13Z'
      fieldsType: FieldsV1
      fieldsV1:
        'f:metadata':
          'f:annotations':
            'f:status': {}
    - manager: kubelet
      operation: Update
      apiVersion: v1
      time: '2023-04-05T19:09:18Z'
      fieldsType: FieldsV1
      fieldsV1:
        'f:status':
          'f:conditions':
            'k:{"type":"ContainersReady"}':
              .: {}
              'f:lastProbeTime': {}
              'f:lastTransitionTime': {}
              'f:message': {}
              'f:reason': {}
              'f:status': {}
              'f:type': {}
            'k:{"type":"Initialized"}':
              .: {}
              'f:lastProbeTime': {}
              'f:lastTransitionTime': {}
              'f:status': {}
              'f:type': {}
            'k:{"type":"Ready"}':
              .: {}
              'f:lastProbeTime': {}
              'f:lastTransitionTime': {}
              'f:message': {}
              'f:reason': {}
              'f:status': {}
              'f:type': {}
          'f:containerStatuses': {}
          'f:hostIP': {}
          'f:phase': {}
          'f:podIP': {}
          'f:podIPs':
            .: {}
            'k:{"ip":"10.129.2.169"}':
              .: {}
              'f:ip': {}
          'f:startTime': {}
      subresource: status
  namespace: pgo
  ownerReferences:
    - apiVersion: apps/v1
      kind: ReplicaSet
      name: dev-eng-pg-cluster-99665545b
      uid: 1dcd828f-1630-46af-a3b6-611580ffb292
      controller: true
      blockOwnerDeletion: true
  labels:
    pgouser: admin
    pgo-version: 1.4.0
    service-name: dev-eng-pg-cluster
    pg-cluster: dev-eng-pg-cluster
    vendor: crunchydata
    name: dev-eng-pg-cluster
    deployment-name: dev-eng-pg-cluster
    pg-pod-anti-affinity: preferred
    crunchy-pgha-scope: dev-eng-pg-cluster
    pod-template-hash: 99665545b
    pgo-pg-database: 'true'
spec:
  restartPolicy: Always
  serviceAccountName: pgo-pg
  imagePullSecrets:
    - name: postgres-operator-dockercfg-f7xj9
    - name: pgo-pg-dockercfg-dmhw6
  priority: 0
  schedulerName: default-scheduler
  enableServiceLinks: true
  affinity:
    podAntiAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
        - weight: 1
          podAffinityTerm:
            labelSelector:
              matchExpressions:
                - key: vendor
                  operator: In
                  values:
                    - crunchydata
                - key: pg-pod-anti-affinity
                  operator: Exists
                - key: pg-cluster
                  operator: In
                  values:
                    - dev-eng-pg-cluster
            topologyKey: kubernetes.io/hostname
  terminationGracePeriodSeconds: 30
  preemptionPolicy: PreemptLowerPriority
  nodeName: dev-eng-qccd4-worker-0-s6qxn
  securityContext:
    seLinuxOptions:
      level: 's0:c30,c25'
    supplementalGroups:
      - 1001
  containers:
    - resources:
        limits:
          cpu: '6'
          memory: 12Gi
        requests:
          cpu: '1'
          memory: 256Mi
      readinessProbe:
        exec:
          command:
            - /opt/crunchy/bin/postgres-ha/health/pgha-readiness.sh
        initialDelaySeconds: 15
        timeoutSeconds: 1
        periodSeconds: 10
        successThreshold: 1
        failureThreshold: 3
      terminationMessagePath: /dev/termination-log
      name: database
      livenessProbe:
        exec:
          command:
            - /opt/crunchy/bin/postgres-ha/health/pgha-liveness.sh
        initialDelaySeconds: 30
        timeoutSeconds: 10
        periodSeconds: 15
        successThreshold: 1
        failureThreshold: 3
      env:
        - name: MODE
          value: postgres
        - name: PGHA_PG_PORT
          value: '5432'
        - name: PGHA_USER
          value: postgres
        - name: PGHA_INIT
          valueFrom:
            configMapKeyRef:
              name: dev-eng-pg-cluster-pgha-config
              key: init
        - name: PATRONI_POSTGRESQL_DATA_DIR
          value: /pgdata/dev-eng-pg-cluster
        - name: PGBACKREST_REPO1_S3_BUCKET
          value: hart-okd-backups
        - name: PGBACKREST_REPO1_S3_ENDPOINT
          value: s3.us-west-2.amazonaws.com
        - name: PGBACKREST_REPO1_S3_REGION
          value: us-west-2
        - name: PGBACKREST_REPO1_S3_KEY
          valueFrom:
            secretKeyRef:
              name: dev-eng-pg-cluster-backrest-repo-config
              key: aws-s3-key
        - name: PGBACKREST_REPO1_S3_KEY_SECRET
          valueFrom:
            secretKeyRef:
              name: dev-eng-pg-cluster-backrest-repo-config
              key: aws-s3-key-secret
        - name: PGBACKREST_REPO1_S3_CA_FILE
          value: /sshd/aws-s3-ca.crt
        - name: PGBACKREST_REPO1_HOST_CMD
          value: /usr/local/bin/archive-push-s3.sh
        - name: PGBACKREST_REPO1_S3_URI_STYLE
          value: path
        - name: PGHA_PGBACKREST_S3_VERIFY_TLS
          value: 'false'
        - name: PGBACKREST_STANZA
          value: db
        - name: PGBACKREST_REPO1_HOST
          value: dev-eng-pg-cluster-backrest-shared-repo
        - name: BACKREST_SKIP_CREATE_STANZA
          value: 'true'
        - name: PGHA_PGBACKREST
          value: 'true'
        - name: PGBACKREST_REPO1_PATH
          value: /dev-eng/postgres-cluster/
        - name: PGBACKREST_DB_PATH
          value: /pgdata/dev-eng-pg-cluster
        - name: ENABLE_SSHD
          value: 'true'
        - name: PGBACKREST_LOG_PATH
          value: /tmp
        - name: PGBACKREST_PG1_SOCKET_PATH
          value: /tmp
        - name: PGBACKREST_PG1_PORT
          value: '5432'
        - name: PGBACKREST_REPO1_TYPE
          value: posix
        - name: PGHA_PGBACKREST_LOCAL_S3_STORAGE
          value: 'true'
        - name: PGHA_PGBACKREST_LOCAL_GCS_STORAGE
          value: 'false'
        - name: PGHA_DATABASE
          value: pgdb
        - name: PGHA_REPLICA_REINIT_ON_START_FAIL
          value: 'true'
        - name: PGHA_SYNC_REPLICATION
          value: 'false'
        - name: PGHA_TLS_ENABLED
          value: 'true'
        - name: PGHA_TLS_ONLY
          value: 'false'
        - name: PGHA_PASSWORD_TYPE
        - name: PGHA_STANDBY
          value: 'false'
        - name: PATRONI_KUBERNETES_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        - name: PATRONI_KUBERNETES_SCOPE_LABEL
          value: crunchy-pgha-scope
        - name: PATRONI_SCOPE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: 'metadata.labels[''crunchy-pgha-scope'']'
        - name: PATRONI_KUBERNETES_LABELS
          value: '{vendor: "crunchydata"}'
        - name: PATRONI_LOG_LEVEL
          value: INFO
        - name: PGHOST
          value: /tmp
        - name: LD_PRELOAD
          value: /usr/lib64/libnss_wrapper.so
        - name: NSS_WRAPPER_PASSWD
          value: /tmp/nss_wrapper/postgres/passwd
        - name: NSS_WRAPPER_GROUP
          value: /tmp/nss_wrapper/postgres/group
      securityContext:
        capabilities:
          drop:
            - MKNOD
        privileged: false
        readOnlyRootFilesystem: true
        allowPrivilegeEscalation: false
      ports:
        - name: postgres
          containerPort: 5432
          protocol: TCP
        - name: patroni
          containerPort: 8009
          protocol: TCP
      imagePullPolicy: IfNotPresent
      volumeMounts:
        - name: pgdata
          mountPath: /pgdata
        - name: user-volume
          mountPath: /pgconf/pguser
        - name: primary-volume
          mountPath: /pgconf/pgreplicator
        - name: root-volume
          mountPath: /pgconf/pgsuper
        - name: tls-server
          mountPath: /pgconf/tls
        - name: tls-replication
          mountPath: /pgconf/tls-replication
        - name: sshd
          readOnly: true
          mountPath: /sshd
        - name: ssh-config
          readOnly: true
          mountPath: /etc/ssh
        - name: pgconf-volume
          mountPath: /pgconf
        - name: dshm
          mountPath: /dev/shm
        - name: pgbackrest-config
          mountPath: /etc/pgbackrest/conf.d
        - name: podinfo
          mountPath: /etc/podinfo
        - name: tmp
          mountPath: /tmp
        - name: kube-api-access-dzkl8
          readOnly: true
          mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      terminationMessagePolicy: File
      image: 'percona/percona-postgresql-operator:1.4.0-ppg14-postgres-ha'
  serviceAccount: pgo-pg
  volumes:
    - name: pgdata
      persistentVolumeClaim:
        claimName: dev-eng-pg-cluster
    - name: user-volume
      secret:
        secretName: dev-eng-pg-cluster-pguser-secret
        defaultMode: 420
    - name: primary-volume
      secret:
        secretName: dev-eng-pg-cluster-primaryuser-secret
        defaultMode: 420
    - name: sshd
      secret:
        secretName: dev-eng-pg-cluster-backrest-repo-config
        defaultMode: 420
    - name: ssh-config
      secret:
        secretName: dev-eng-pg-cluster-backrest-repo-config
        items:
          - key: config
            path: ssh_config
        defaultMode: 420
    - name: root-volume
      secret:
        secretName: dev-eng-pg-cluster-postgres-secret
        defaultMode: 420
    - name: tls-server
      projected:
        sources:
          - secret:
              name: dev-eng-pg-cluster-ssl-keypair
          - secret:
              name: dev-eng-pg-cluster-replication-ssl-keypair
              items:
                - key: tls.key
                  path: tls-replication.key
                - key: tls.crt
                  path: tls-replication.crt
          - secret:
              name: dev-eng-pg-cluster-ssl-ca
        defaultMode: 288
    - name: tls-replication
      emptyDir:
        medium: Memory
        sizeLimit: 2Mi
    - name: report
      emptyDir:
        medium: Memory
        sizeLimit: 64Mi
    - name: dshm
      emptyDir:
        medium: Memory
    - name: tmp
      emptyDir:
        medium: Memory
        sizeLimit: 16Mi
    - name: pgbackrest-config
      projected:
        sources:
          - configMap:
              name: dev-eng-pg-cluster-config-backrest
              optional: true
          - secret:
              name: dev-eng-pg-cluster-config-backrest
              optional: true
        defaultMode: 420
    - name: pgconf-volume
      projected:
        sources:
          - configMap:
              name: dev-eng-pg-cluster-pgha-config
              optional: true
        defaultMode: 420
    - name: podinfo
      downwardAPI:
        items:
          - path: cpu_limit
            resourceFieldRef:
              containerName: database
              resource: limits.cpu
              divisor: 1m
          - path: cpu_request
            resourceFieldRef:
              containerName: database
              resource: requests.cpu
              divisor: 1m
          - path: mem_limit
            resourceFieldRef:
              containerName: database
              resource: limits.memory
              divisor: '0'
          - path: mem_request
            resourceFieldRef:
              containerName: database
              resource: requests.memory
              divisor: '0'
          - path: labels
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.labels
          - path: annotations
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.annotations
        defaultMode: 420
    - name: kube-api-access-dzkl8
      projected:
        sources:
          - serviceAccountToken:
              expirationSeconds: 3607
              path: token
          - configMap:
              name: kube-root-ca.crt
              items:
                - key: ca.crt
                  path: ca.crt
          - downwardAPI:
              items:
                - path: namespace
                  fieldRef:
                    apiVersion: v1
                    fieldPath: metadata.namespace
          - configMap:
              name: openshift-service-ca.crt
              items:
                - key: service-ca.crt
                  path: service-ca.crt
        defaultMode: 420
  dnsPolicy: ClusterFirst
  tolerations:
    - key: node.kubernetes.io/not-ready
      operator: Exists
      effect: NoExecute
      tolerationSeconds: 300
    - key: node.kubernetes.io/unreachable
      operator: Exists
      effect: NoExecute
      tolerationSeconds: 300
    - key: node.kubernetes.io/memory-pressure
      operator: Exists
      effect: NoSchedule
status:
  phase: Running
  conditions:
    - type: Initialized
      status: 'True'
      lastProbeTime: null
      lastTransitionTime: '2023-04-05T19:08:59Z'
    - type: Ready
      status: 'False'
      lastProbeTime: null
      lastTransitionTime: '2023-04-05T19:08:59Z'
      reason: ContainersNotReady
      message: 'containers with unready status: [database]'
    - type: ContainersReady
      status: 'False'
      lastProbeTime: null
      lastTransitionTime: '2023-04-05T19:08:59Z'
      reason: ContainersNotReady
      message: 'containers with unready status: [database]'
    - type: PodScheduled
      status: 'True'
      lastProbeTime: null
      lastTransitionTime: '2023-04-05T19:08:59Z'
  hostIP: 10.0.1.154
  podIP: 10.129.2.169
  podIPs:
    - ip: 10.129.2.169
  startTime: '2023-04-05T19:08:59Z'
  containerStatuses:
    - restartCount: 1
      started: false
      ready: false
      name: database
      state:
        waiting:
          reason: CrashLoopBackOff
          message: >-
            back-off 10s restarting failed container=database
            pod=dev-eng-pg-cluster-99665545b-6txbx_pgo(f377f440-0f7e-45e3-ad39-854b7ac96f15)
      imageID: >-
        docker.io/percona/percona-postgresql-operator@sha256:93b69b07914f1ed43812013b7bfc75692369bc39fc24349853584165fc192e45
      image: 'docker.io/percona/percona-postgresql-operator:1.4.0-ppg14-postgres-ha'
      lastState:
        terminated:
          exitCode: 0
          reason: Completed
          startedAt: '2023-04-05T19:09:15Z'
          finishedAt: '2023-04-05T19:09:17Z'
          containerID: >-
            cri-o://4d110307223888180138d2a48e3d4804953f7180921490e9e55eb6cd2d58558c
      containerID: 'cri-o://4d110307223888180138d2a48e3d4804953f7180921490e9e55eb6cd2d58558c'
  qosClass: Burstable

Ok

It looks like you need to set disable_fsgroup: false as it is stated in Install on OpenShift - Percona Operator for PostgreSQL operator installation doc.
Please reinstall the operator and try to redeploy the cluster

Hello,
If you are referring to the following line, I made sure to do that initially. Just in case I below away the namspace and started over again with the same issue.

sed -i ‘/disable_auto_failover: “false”/a \ \ \ \ disable_fsgroup: “false”’ deploy/operator.yaml

kind: Pod
apiVersion: v1
metadata:
  generateName: dev-eng-pg-cluster-99665545b-
  annotations:
    k8s.ovn.org/pod-networks: >-
      {"default":{"ip_addresses":["10.129.2.174/23"],"mac_address":"0a:58:0a:81:02:ae","gateway_ips":["10.129.2.1"],"ip_address":"10.129.2.174/23","gateway_ip":"10.129.2.1"}}
    k8s.v1.cni.cncf.io/network-status: |-
      [{
          "name": "ovn-kubernetes",
          "interface": "eth0",
          "ips": [
              "10.129.2.174"
          ],
          "mac": "0a:58:0a:81:02:ae",
          "default": true,
          "dns": {}
      }]
    k8s.v1.cni.cncf.io/networks-status: |-
      [{
          "name": "ovn-kubernetes",
          "interface": "eth0",
          "ips": [
              "10.129.2.174"
          ],
          "mac": "0a:58:0a:81:02:ae",
          "default": true,
          "dns": {}
      }]
    keep-backups: 'true'
    keep-data: 'true'
    openshift.io/scc: anyuid
    status: >-
      {"conn_url":"postgres://10.129.2.174:5432/postgres","api_url":"http://10.129.2.174:8009/patroni","state":"stopped","role":"uninitialized","version":"2.1.4"}
  resourceVersion: '32470037'
  name: dev-eng-pg-cluster-99665545b-hp4gv
  uid: c8946c7b-b216-41ca-b64d-5fe83fa4aeda
  creationTimestamp: '2023-04-05T19:37:24Z'
  managedFields:
    - manager: Go-http-client
      operation: Update
      apiVersion: v1
      time: '2023-04-05T19:37:24Z'
      fieldsType: FieldsV1
      fieldsV1:
        'f:metadata':
          'f:labels':
            'f:service-name': {}
    - manager: kube-controller-manager
      operation: Update
      apiVersion: v1
      time: '2023-04-05T19:37:24Z'
      fieldsType: FieldsV1
      fieldsV1:
        'f:metadata':
          'f:annotations':
            .: {}
            'f:keep-backups': {}
            'f:keep-data': {}
          'f:generateName': {}
          'f:labels':
            'f:pod-template-hash': {}
            'f:pgo-pg-database': {}
            'f:pg-pod-anti-affinity': {}
            'f:crunchy-pgha-scope': {}
            .: {}
            'f:pgo-version': {}
            'f:pgouser': {}
            'f:pg-cluster': {}
            'f:vendor': {}
            'f:name': {}
            'f:deployment-name': {}
          'f:ownerReferences':
            .: {}
            'k:{"uid":"9d53aff3-3453-4ea2-af77-f5d3e8db26ed"}': {}
        'f:spec':
          'f:volumes':
            'k:{"name":"sshd"}':
              .: {}
              'f:name': {}
              'f:secret':
                .: {}
                'f:defaultMode': {}
                'f:secretName': {}
            'k:{"name":"pgdata"}':
              .: {}
              'f:name': {}
              'f:persistentVolumeClaim':
                .: {}
                'f:claimName': {}
            'k:{"name":"podinfo"}':
              .: {}
              'f:downwardAPI':
                .: {}
                'f:defaultMode': {}
                'f:items': {}
              'f:name': {}
            'k:{"name":"primary-volume"}':
              .: {}
              'f:name': {}
              'f:secret':
                .: {}
                'f:defaultMode': {}
                'f:secretName': {}
            'k:{"name":"tmp"}':
              .: {}
              'f:emptyDir':
                .: {}
                'f:medium': {}
                'f:sizeLimit': {}
              'f:name': {}
            'k:{"name":"ssh-config"}':
              .: {}
              'f:name': {}
              'f:secret':
                .: {}
                'f:defaultMode': {}
                'f:items': {}
                'f:secretName': {}
            'k:{"name":"tls-replication"}':
              .: {}
              'f:emptyDir':
                .: {}
                'f:medium': {}
                'f:sizeLimit': {}
              'f:name': {}
            .: {}
            'k:{"name":"user-volume"}':
              .: {}
              'f:name': {}
              'f:secret':
                .: {}
                'f:defaultMode': {}
                'f:secretName': {}
            'k:{"name":"pgbackrest-config"}':
              .: {}
              'f:name': {}
              'f:projected':
                .: {}
                'f:defaultMode': {}
                'f:sources': {}
            'k:{"name":"root-volume"}':
              .: {}
              'f:name': {}
              'f:secret':
                .: {}
                'f:defaultMode': {}
                'f:secretName': {}
            'k:{"name":"dshm"}':
              .: {}
              'f:emptyDir':
                .: {}
                'f:medium': {}
              'f:name': {}
            'k:{"name":"report"}':
              .: {}
              'f:emptyDir':
                .: {}
                'f:medium': {}
                'f:sizeLimit': {}
              'f:name': {}
            'k:{"name":"tls-server"}':
              .: {}
              'f:name': {}
              'f:projected':
                .: {}
                'f:defaultMode': {}
                'f:sources': {}
            'k:{"name":"pgconf-volume"}':
              .: {}
              'f:name': {}
              'f:projected':
                .: {}
                'f:defaultMode': {}
                'f:sources': {}
          'f:containers':
            'k:{"name":"database"}':
              'f:image': {}
              'f:volumeMounts':
                'k:{"mountPath":"/pgconf"}':
                  .: {}
                  'f:mountPath': {}
                  'f:name': {}
                'k:{"mountPath":"/pgconf/pguser"}':
                  .: {}
                  'f:mountPath': {}
                  'f:name': {}
                'k:{"mountPath":"/sshd"}':
                  .: {}
                  'f:mountPath': {}
                  'f:name': {}
                  'f:readOnly': {}
                'k:{"mountPath":"/tmp"}':
                  .: {}
                  'f:mountPath': {}
                  'f:name': {}
                'k:{"mountPath":"/etc/podinfo"}':
                  .: {}
                  'f:mountPath': {}
                  'f:name': {}
                'k:{"mountPath":"/pgdata"}':
                  .: {}
                  'f:mountPath': {}
                  'f:name': {}
                'k:{"mountPath":"/etc/pgbackrest/conf.d"}':
                  .: {}
                  'f:mountPath': {}
                  'f:name': {}
                'k:{"mountPath":"/dev/shm"}':
                  .: {}
                  'f:mountPath': {}
                  'f:name': {}
                .: {}
                'k:{"mountPath":"/pgconf/tls-replication"}':
                  .: {}
                  'f:mountPath': {}
                  'f:name': {}
                'k:{"mountPath":"/pgconf/tls"}':
                  .: {}
                  'f:mountPath': {}
                  'f:name': {}
                'k:{"mountPath":"/pgconf/pgreplicator"}':
                  .: {}
                  'f:mountPath': {}
                  'f:name': {}
                'k:{"mountPath":"/pgconf/pgsuper"}':
                  .: {}
                  'f:mountPath': {}
                  'f:name': {}
                'k:{"mountPath":"/etc/ssh"}':
                  .: {}
                  'f:mountPath': {}
                  'f:name': {}
                  'f:readOnly': {}
              'f:terminationMessagePolicy': {}
              .: {}
              'f:resources':
                .: {}
                'f:limits':
                  .: {}
                  'f:cpu': {}
                  'f:memory': {}
                'f:requests':
                  .: {}
                  'f:cpu': {}
                  'f:memory': {}
              'f:livenessProbe':
                .: {}
                'f:exec':
                  .: {}
                  'f:command': {}
                'f:failureThreshold': {}
                'f:initialDelaySeconds': {}
                'f:periodSeconds': {}
                'f:successThreshold': {}
                'f:timeoutSeconds': {}
              'f:env':
                'k:{"name":"PGBACKREST_REPO1_S3_BUCKET"}':
                  .: {}
                  'f:name': {}
                  'f:value': {}
                'k:{"name":"PGBACKREST_REPO1_HOST_CMD"}':
                  .: {}
                  'f:name': {}
                  'f:value': {}
                'k:{"name":"PATRONI_SCOPE"}':
                  .: {}
                  'f:name': {}
                  'f:valueFrom':
                    .: {}
                    'f:fieldRef': {}
                'k:{"name":"PGHA_STANDBY"}':
                  .: {}
                  'f:name': {}
                  'f:value': {}
                'k:{"name":"PGBACKREST_PG1_SOCKET_PATH"}':
                  .: {}
                  'f:name': {}
                  'f:value': {}
                'k:{"name":"PGHOST"}':
                  .: {}
                  'f:name': {}
                  'f:value': {}
                'k:{"name":"PGBACKREST_REPO1_PATH"}':
                  .: {}
                  'f:name': {}
                  'f:value': {}
                'k:{"name":"PGBACKREST_PG1_PORT"}':
                  .: {}
                  'f:name': {}
                  'f:value': {}
                'k:{"name":"PATRONI_KUBERNETES_LABELS"}':
                  .: {}
                  'f:name': {}
                  'f:value': {}
                'k:{"name":"PGBACKREST_REPO1_S3_KEY_SECRET"}':
                  .: {}
                  'f:name': {}
                  'f:valueFrom':
                    .: {}
                    'f:secretKeyRef': {}
                'k:{"name":"LD_PRELOAD"}':
                  .: {}
                  'f:name': {}
                  'f:value': {}
                'k:{"name":"MODE"}':
                  .: {}
                  'f:name': {}
                  'f:value': {}
                'k:{"name":"PGBACKREST_REPO1_HOST"}':
                  .: {}
                  'f:name': {}
                  'f:value': {}
                'k:{"name":"PGHA_REPLICA_REINIT_ON_START_FAIL"}':
                  .: {}
                  'f:name': {}
                  'f:value': {}
                'k:{"name":"PGHA_PGBACKREST_S3_VERIFY_TLS"}':
                  .: {}
                  'f:name': {}
                  'f:value': {}
                'k:{"name":"PGHA_SYNC_REPLICATION"}':
                  .: {}
                  'f:name': {}
                  'f:value': {}
                'k:{"name":"PGBACKREST_REPO1_S3_REGION"}':
                  .: {}
                  'f:name': {}
                  'f:value': {}
                'k:{"name":"PATRONI_KUBERNETES_NAMESPACE"}':
                  .: {}
                  'f:name': {}
                  'f:valueFrom':
                    .: {}
                    'f:fieldRef': {}
                'k:{"name":"ENABLE_SSHD"}':
                  .: {}
                  'f:name': {}
                  'f:value': {}
                'k:{"name":"PGBACKREST_REPO1_S3_ENDPOINT"}':
                  .: {}
                  'f:name': {}
                  'f:value': {}
                'k:{"name":"PGHA_USER"}':
                  .: {}
                  'f:name': {}
                  'f:value': {}
                .: {}
                'k:{"name":"BACKREST_SKIP_CREATE_STANZA"}':
                  .: {}
                  'f:name': {}
                  'f:value': {}
                'k:{"name":"PGHA_INIT"}':
                  .: {}
                  'f:name': {}
                  'f:valueFrom':
                    .: {}
                    'f:configMapKeyRef': {}
                'k:{"name":"PGHA_TLS_ONLY"}':
                  .: {}
                  'f:name': {}
                  'f:value': {}
                'k:{"name":"PATRONI_POSTGRESQL_DATA_DIR"}':
                  .: {}
                  'f:name': {}
                  'f:value': {}
                'k:{"name":"PGHA_PGBACKREST_LOCAL_S3_STORAGE"}':
                  .: {}
                  'f:name': {}
                  'f:value': {}
                'k:{"name":"PGHA_TLS_ENABLED"}':
                  .: {}
                  'f:name': {}
                  'f:value': {}
                'k:{"name":"PGHA_PGBACKREST_LOCAL_GCS_STORAGE"}':
                  .: {}
                  'f:name': {}
                  'f:value': {}
                'k:{"name":"PGHA_PGBACKREST"}':
                  .: {}
                  'f:name': {}
                  'f:value': {}
                'k:{"name":"PGHA_PASSWORD_TYPE"}':
                  .: {}
                  'f:name': {}
                'k:{"name":"PGBACKREST_STANZA"}':
                  .: {}
                  'f:name': {}
                  'f:value': {}
                'k:{"name":"NSS_WRAPPER_GROUP"}':
                  .: {}
                  'f:name': {}
                  'f:value': {}
                'k:{"name":"PATRONI_LOG_LEVEL"}':
                  .: {}
                  'f:name': {}
                  'f:value': {}
                'k:{"name":"PATRONI_KUBERNETES_SCOPE_LABEL"}':
                  .: {}
                  'f:name': {}
                  'f:value': {}
                'k:{"name":"PGBACKREST_REPO1_TYPE"}':
                  .: {}
                  'f:name': {}
                  'f:value': {}
                'k:{"name":"PGHA_DATABASE"}':
                  .: {}
                  'f:name': {}
                  'f:value': {}
                'k:{"name":"PGBACKREST_REPO1_S3_CA_FILE"}':
                  .: {}
                  'f:name': {}
                  'f:value': {}
                'k:{"name":"PGBACKREST_DB_PATH"}':
                  .: {}
                  'f:name': {}
                  'f:value': {}
                'k:{"name":"PGBACKREST_REPO1_S3_URI_STYLE"}':
                  .: {}
                  'f:name': {}
                  'f:value': {}
                'k:{"name":"PGHA_PG_PORT"}':
                  .: {}
                  'f:name': {}
                  'f:value': {}
                'k:{"name":"PGBACKREST_REPO1_S3_KEY"}':
                  .: {}
                  'f:name': {}
                  'f:valueFrom':
                    .: {}
                    'f:secretKeyRef': {}
                'k:{"name":"NSS_WRAPPER_PASSWD"}':
                  .: {}
                  'f:name': {}
                  'f:value': {}
                'k:{"name":"PGBACKREST_LOG_PATH"}':
                  .: {}
                  'f:name': {}
                  'f:value': {}
              'f:readinessProbe':
                .: {}
                'f:exec':
                  .: {}
                  'f:command': {}
                'f:failureThreshold': {}
                'f:initialDelaySeconds': {}
                'f:periodSeconds': {}
                'f:successThreshold': {}
                'f:timeoutSeconds': {}
              'f:securityContext':
                .: {}
                'f:allowPrivilegeEscalation': {}
                'f:privileged': {}
                'f:readOnlyRootFilesystem': {}
              'f:terminationMessagePath': {}
              'f:imagePullPolicy': {}
              'f:ports':
                .: {}
                'k:{"containerPort":5432,"protocol":"TCP"}':
                  .: {}
                  'f:containerPort': {}
                  'f:name': {}
                  'f:protocol': {}
                'k:{"containerPort":8009,"protocol":"TCP"}':
                  .: {}
                  'f:containerPort': {}
                  'f:name': {}
                  'f:protocol': {}
              'f:name': {}
          'f:dnsPolicy': {}
          'f:serviceAccount': {}
          'f:restartPolicy': {}
          'f:schedulerName': {}
          'f:terminationGracePeriodSeconds': {}
          'f:serviceAccountName': {}
          'f:enableServiceLinks': {}
          'f:securityContext':
            .: {}
            'f:supplementalGroups': {}
          'f:affinity':
            .: {}
            'f:podAntiAffinity':
              .: {}
              'f:preferredDuringSchedulingIgnoredDuringExecution': {}
    - manager: dev-eng-qccd4-master-1.novalocal
      operation: Update
      apiVersion: v1
      time: '2023-04-05T19:37:25Z'
      fieldsType: FieldsV1
      fieldsV1:
        'f:metadata':
          'f:annotations':
            'f:k8s.ovn.org/pod-networks': {}
    - manager: multus
      operation: Update
      apiVersion: v1
      time: '2023-04-05T19:37:35Z'
      fieldsType: FieldsV1
      fieldsV1:
        'f:metadata':
          'f:annotations':
            'f:k8s.v1.cni.cncf.io/network-status': {}
            'f:k8s.v1.cni.cncf.io/networks-status': {}
      subresource: status
    - manager: Patroni
      operation: Update
      apiVersion: v1
      time: '2023-04-05T19:37:37Z'
      fieldsType: FieldsV1
      fieldsV1:
        'f:metadata':
          'f:annotations':
            'f:status': {}
    - manager: kubelet
      operation: Update
      apiVersion: v1
      time: '2023-04-05T19:37:41Z'
      fieldsType: FieldsV1
      fieldsV1:
        'f:status':
          'f:conditions':
            'k:{"type":"ContainersReady"}':
              .: {}
              'f:lastProbeTime': {}
              'f:lastTransitionTime': {}
              'f:message': {}
              'f:reason': {}
              'f:status': {}
              'f:type': {}
            'k:{"type":"Initialized"}':
              .: {}
              'f:lastProbeTime': {}
              'f:lastTransitionTime': {}
              'f:status': {}
              'f:type': {}
            'k:{"type":"Ready"}':
              .: {}
              'f:lastProbeTime': {}
              'f:lastTransitionTime': {}
              'f:message': {}
              'f:reason': {}
              'f:status': {}
              'f:type': {}
          'f:containerStatuses': {}
          'f:hostIP': {}
          'f:phase': {}
          'f:podIP': {}
          'f:podIPs':
            .: {}
            'k:{"ip":"10.129.2.174"}':
              .: {}
              'f:ip': {}
          'f:startTime': {}
      subresource: status
  namespace: pgo
  ownerReferences:
    - apiVersion: apps/v1
      kind: ReplicaSet
      name: dev-eng-pg-cluster-99665545b
      uid: 9d53aff3-3453-4ea2-af77-f5d3e8db26ed
      controller: true
      blockOwnerDeletion: true
  labels:
    pgouser: admin
    pgo-version: 1.4.0
    service-name: dev-eng-pg-cluster
    pg-cluster: dev-eng-pg-cluster
    vendor: crunchydata
    name: dev-eng-pg-cluster
    deployment-name: dev-eng-pg-cluster
    pg-pod-anti-affinity: preferred
    crunchy-pgha-scope: dev-eng-pg-cluster
    pod-template-hash: 99665545b
    pgo-pg-database: 'true'
spec:
  restartPolicy: Always
  serviceAccountName: pgo-pg
  imagePullSecrets:
    - name: postgres-operator-dockercfg-5k8t8
    - name: pgo-pg-dockercfg-5mm5p
  priority: 0
  schedulerName: default-scheduler
  enableServiceLinks: true
  affinity:
    podAntiAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
        - weight: 1
          podAffinityTerm:
            labelSelector:
              matchExpressions:
                - key: vendor
                  operator: In
                  values:
                    - crunchydata
                - key: pg-pod-anti-affinity
                  operator: Exists
                - key: pg-cluster
                  operator: In
                  values:
                    - dev-eng-pg-cluster
            topologyKey: kubernetes.io/hostname
  terminationGracePeriodSeconds: 30
  preemptionPolicy: PreemptLowerPriority
  nodeName: dev-eng-qccd4-worker-0-s6qxn
  securityContext:
    seLinuxOptions:
      level: 's0:c27,c4'
    supplementalGroups:
      - 1001
  containers:
    - resources:
        limits:
          cpu: '6'
          memory: 12Gi
        requests:
          cpu: '1'
          memory: 256Mi
      readinessProbe:
        exec:
          command:
            - /opt/crunchy/bin/postgres-ha/health/pgha-readiness.sh
        initialDelaySeconds: 15
        timeoutSeconds: 1
        periodSeconds: 10
        successThreshold: 1
        failureThreshold: 3
      terminationMessagePath: /dev/termination-log
      name: database
      livenessProbe:
        exec:
          command:
            - /opt/crunchy/bin/postgres-ha/health/pgha-liveness.sh
        initialDelaySeconds: 30
        timeoutSeconds: 10
        periodSeconds: 15
        successThreshold: 1
        failureThreshold: 3
      env:
        - name: MODE
          value: postgres
        - name: PGHA_PG_PORT
          value: '5432'
        - name: PGHA_USER
          value: postgres
        - name: PGHA_INIT
          valueFrom:
            configMapKeyRef:
              name: dev-eng-pg-cluster-pgha-config
              key: init
        - name: PATRONI_POSTGRESQL_DATA_DIR
          value: /pgdata/dev-eng-pg-cluster
        - name: PGBACKREST_REPO1_S3_BUCKET
          value: hart-okd-backups
        - name: PGBACKREST_REPO1_S3_ENDPOINT
          value: s3.us-west-2.amazonaws.com
        - name: PGBACKREST_REPO1_S3_REGION
          value: us-west-2
        - name: PGBACKREST_REPO1_S3_KEY
          valueFrom:
            secretKeyRef:
              name: dev-eng-pg-cluster-backrest-repo-config
              key: aws-s3-key
        - name: PGBACKREST_REPO1_S3_KEY_SECRET
          valueFrom:
            secretKeyRef:
              name: dev-eng-pg-cluster-backrest-repo-config
              key: aws-s3-key-secret
        - name: PGBACKREST_REPO1_S3_CA_FILE
          value: /sshd/aws-s3-ca.crt
        - name: PGBACKREST_REPO1_HOST_CMD
          value: /usr/local/bin/archive-push-s3.sh
        - name: PGBACKREST_REPO1_S3_URI_STYLE
          value: path
        - name: PGHA_PGBACKREST_S3_VERIFY_TLS
          value: 'false'
        - name: PGBACKREST_STANZA
          value: db
        - name: PGBACKREST_REPO1_HOST
          value: dev-eng-pg-cluster-backrest-shared-repo
        - name: BACKREST_SKIP_CREATE_STANZA
          value: 'true'
        - name: PGHA_PGBACKREST
          value: 'true'
        - name: PGBACKREST_REPO1_PATH
          value: /dev-eng/postgres-cluster/
        - name: PGBACKREST_DB_PATH
          value: /pgdata/dev-eng-pg-cluster
        - name: ENABLE_SSHD
          value: 'true'
        - name: PGBACKREST_LOG_PATH
          value: /tmp
        - name: PGBACKREST_PG1_SOCKET_PATH
          value: /tmp
        - name: PGBACKREST_PG1_PORT
          value: '5432'
        - name: PGBACKREST_REPO1_TYPE
          value: posix
        - name: PGHA_PGBACKREST_LOCAL_S3_STORAGE
          value: 'true'
        - name: PGHA_PGBACKREST_LOCAL_GCS_STORAGE
          value: 'false'
        - name: PGHA_DATABASE
          value: pgdb
        - name: PGHA_REPLICA_REINIT_ON_START_FAIL
          value: 'true'
        - name: PGHA_SYNC_REPLICATION
          value: 'false'
        - name: PGHA_TLS_ENABLED
          value: 'true'
        - name: PGHA_TLS_ONLY
          value: 'false'
        - name: PGHA_PASSWORD_TYPE
        - name: PGHA_STANDBY
          value: 'false'
        - name: PATRONI_KUBERNETES_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        - name: PATRONI_KUBERNETES_SCOPE_LABEL
          value: crunchy-pgha-scope
        - name: PATRONI_SCOPE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: 'metadata.labels[''crunchy-pgha-scope'']'
        - name: PATRONI_KUBERNETES_LABELS
          value: '{vendor: "crunchydata"}'
        - name: PATRONI_LOG_LEVEL
          value: INFO
        - name: PGHOST
          value: /tmp
        - name: LD_PRELOAD
          value: /usr/lib64/libnss_wrapper.so
        - name: NSS_WRAPPER_PASSWD
          value: /tmp/nss_wrapper/postgres/passwd
        - name: NSS_WRAPPER_GROUP
          value: /tmp/nss_wrapper/postgres/group
      securityContext:
        capabilities:
          drop:
            - MKNOD
        privileged: false
        readOnlyRootFilesystem: true
        allowPrivilegeEscalation: false
      ports:
        - name: postgres
          containerPort: 5432
          protocol: TCP
        - name: patroni
          containerPort: 8009
          protocol: TCP
      imagePullPolicy: IfNotPresent
      volumeMounts:
        - name: pgdata
          mountPath: /pgdata
        - name: user-volume
          mountPath: /pgconf/pguser
        - name: primary-volume
          mountPath: /pgconf/pgreplicator
        - name: root-volume
          mountPath: /pgconf/pgsuper
        - name: tls-server
          mountPath: /pgconf/tls
        - name: tls-replication
          mountPath: /pgconf/tls-replication
        - name: sshd
          readOnly: true
          mountPath: /sshd
        - name: ssh-config
          readOnly: true
          mountPath: /etc/ssh
        - name: pgconf-volume
          mountPath: /pgconf
        - name: dshm
          mountPath: /dev/shm
        - name: pgbackrest-config
          mountPath: /etc/pgbackrest/conf.d
        - name: podinfo
          mountPath: /etc/podinfo
        - name: tmp
          mountPath: /tmp
        - name: kube-api-access-4g9zx
          readOnly: true
          mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      terminationMessagePolicy: File
      image: 'percona/percona-postgresql-operator:1.4.0-ppg14-postgres-ha'
  serviceAccount: pgo-pg
  volumes:
    - name: pgdata
      persistentVolumeClaim:
        claimName: dev-eng-pg-cluster
    - name: user-volume
      secret:
        secretName: dev-eng-pg-cluster-pguser-secret
        defaultMode: 420
    - name: primary-volume
      secret:
        secretName: dev-eng-pg-cluster-primaryuser-secret
        defaultMode: 420
    - name: sshd
      secret:
        secretName: dev-eng-pg-cluster-backrest-repo-config
        defaultMode: 420
    - name: ssh-config
      secret:
        secretName: dev-eng-pg-cluster-backrest-repo-config
        items:
          - key: config
            path: ssh_config
        defaultMode: 420
    - name: root-volume
      secret:
        secretName: dev-eng-pg-cluster-postgres-secret
        defaultMode: 420
    - name: tls-server
      projected:
        sources:
          - secret:
              name: dev-eng-pg-cluster-ssl-keypair
          - secret:
              name: dev-eng-pg-cluster-replication-ssl-keypair
              items:
                - key: tls.key
                  path: tls-replication.key
                - key: tls.crt
                  path: tls-replication.crt
          - secret:
              name: dev-eng-pg-cluster-ssl-ca
        defaultMode: 288
    - name: tls-replication
      emptyDir:
        medium: Memory
        sizeLimit: 2Mi
    - name: report
      emptyDir:
        medium: Memory
        sizeLimit: 64Mi
    - name: dshm
      emptyDir:
        medium: Memory
    - name: tmp
      emptyDir:
        medium: Memory
        sizeLimit: 16Mi
    - name: pgbackrest-config
      projected:
        sources:
          - configMap:
              name: dev-eng-pg-cluster-config-backrest
              optional: true
          - secret:
              name: dev-eng-pg-cluster-config-backrest
              optional: true
        defaultMode: 420
    - name: pgconf-volume
      projected:
        sources:
          - configMap:
              name: dev-eng-pg-cluster-pgha-config
              optional: true
        defaultMode: 420
    - name: podinfo
      downwardAPI:
        items:
          - path: cpu_limit
            resourceFieldRef:
              containerName: database
              resource: limits.cpu
              divisor: 1m
          - path: cpu_request
            resourceFieldRef:
              containerName: database
              resource: requests.cpu
              divisor: 1m
          - path: mem_limit
            resourceFieldRef:
              containerName: database
              resource: limits.memory
              divisor: '0'
          - path: mem_request
            resourceFieldRef:
              containerName: database
              resource: requests.memory
              divisor: '0'
          - path: labels
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.labels
          - path: annotations
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.annotations
        defaultMode: 420
    - name: kube-api-access-4g9zx
      projected:
        sources:
          - serviceAccountToken:
              expirationSeconds: 3607
              path: token
          - configMap:
              name: kube-root-ca.crt
              items:
                - key: ca.crt
                  path: ca.crt
          - downwardAPI:
              items:
                - path: namespace
                  fieldRef:
                    apiVersion: v1
                    fieldPath: metadata.namespace
          - configMap:
              name: openshift-service-ca.crt
              items:
                - key: service-ca.crt
                  path: service-ca.crt
        defaultMode: 420
  dnsPolicy: ClusterFirst
  tolerations:
    - key: node.kubernetes.io/not-ready
      operator: Exists
      effect: NoExecute
      tolerationSeconds: 300
    - key: node.kubernetes.io/unreachable
      operator: Exists
      effect: NoExecute
      tolerationSeconds: 300
    - key: node.kubernetes.io/memory-pressure
      operator: Exists
      effect: NoSchedule
status:
  phase: Running
  conditions:
    - type: Initialized
      status: 'True'
      lastProbeTime: null
      lastTransitionTime: '2023-04-05T19:37:25Z'
    - type: Ready
      status: 'False'
      lastProbeTime: null
      lastTransitionTime: '2023-04-05T19:37:25Z'
      reason: ContainersNotReady
      message: 'containers with unready status: [database]'
    - type: ContainersReady
      status: 'False'
      lastProbeTime: null
      lastTransitionTime: '2023-04-05T19:37:25Z'
      reason: ContainersNotReady
      message: 'containers with unready status: [database]'
    - type: PodScheduled
      status: 'True'
      lastProbeTime: null
      lastTransitionTime: '2023-04-05T19:37:25Z'
  hostIP: 10.0.1.154
  podIP: 10.129.2.174
  podIPs:
    - ip: 10.129.2.174
  startTime: '2023-04-05T19:37:25Z'
  containerStatuses:
    - restartCount: 1
      started: false
      ready: false
      name: database
      state:
        waiting:
          reason: CrashLoopBackOff
          message: >-
            back-off 10s restarting failed container=database
            pod=dev-eng-pg-cluster-99665545b-hp4gv_pgo(c8946c7b-b216-41ca-b64d-5fe83fa4aeda)
      imageID: >-
        docker.io/percona/percona-postgresql-operator@sha256:93b69b07914f1ed43812013b7bfc75692369bc39fc24349853584165fc192e45
      image: 'docker.io/percona/percona-postgresql-operator:1.4.0-ppg14-postgres-ha'
      lastState:
        terminated:
          exitCode: 0
          reason: Completed
          startedAt: '2023-04-05T19:37:38Z'
          finishedAt: '2023-04-05T19:37:39Z'
          containerID: >-
            cri-o://368f50cf719489ffec8a5ca06acc2e576104156f996ade78aa6a4e24a7d5f06d
      containerID: 'cri-o://368f50cf719489ffec8a5ca06acc2e576104156f996ade78aa6a4e24a7d5f06d'
  qosClass: Burstable
`

I also attached what the operator.yaml file looks like after running the sed command in the instructions:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: pgo-deployer-sa
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: pgo-deployer-cr
rules:
  - apiGroups:
      - ''
    resources:
      - namespaces
    verbs:
      - get
      - list
      - create
      - patch
      - delete
  - apiGroups:
      - ''
    resources:
      - pods
    verbs:
      - list
  - apiGroups:
      - ''
    resources:
      - secrets
    verbs:
      - list
      - get
      - create
      - delete
      - patch
  - apiGroups:
      - ''
    resources:
      - configmaps
      - services
      - persistentvolumeclaims
    verbs:
      - get
      - create
      - delete
      - list
      - patch
  - apiGroups:
      - ''
    resources:
      - serviceaccounts
    verbs:
      - get
      - create
      - delete
      - patch
      - list
  - apiGroups:
      - apps
      - extensions
    resources:
      - deployments
      - replicasets
    verbs:
      - get
      - list
      - watch
      - create
      - delete
  - apiGroups:
      - apiextensions.k8s.io
    resources:
      - customresourcedefinitions
    verbs:
      - get
      - create
      - delete
      - patch
  - apiGroups:
      - rbac.authorization.k8s.io
    resources:
      - clusterroles
      - clusterrolebindings
      - roles
      - rolebindings
    verbs:
      - get
      - create
      - delete
      - bind
      - escalate
  - apiGroups:
      - rbac.authorization.k8s.io
    resources:
      - roles
    verbs:
      - create
      - delete
  - apiGroups:
      - batch
    resources:
      - jobs
    verbs:
      - delete
      - list
  - apiGroups:
      - pg.percona.com
    resources:
      - perconapgclusters
      - perconapgclusters/status
      - pgclusters
      - pgreplicas
      - pgpolicies
      - pgtasks
    verbs:
      - delete
      - list
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: pgo-deployer-cm
data:
  values.yaml: |-
    archive_mode: "true"
    archive_timeout: "60"
    ccp_image_pull_secret: ""
    ccp_image_pull_secret_manifest: ""
    create_rbac: "true"
    delete_operator_namespace: "false"
    delete_watched_namespaces: "false"
    disable_telemetry: "false"
    namespace: "pgo"
    namespace_mode: "disabled"
    pgo_image_prefix: "percona/percona-postgresql-operator"
    pgo_image_pull_policy: "Always"
    pgo_image_pull_secret: ""
    pgo_image_pull_secret_manifest: ""
    pgo_image_tag: "1.4.0"
    pgo_operator_namespace: "pgo"
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: pgo-deployer-crb
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: pgo-deployer-cr
subjects:
  - kind: ServiceAccount
    name: pgo-deployer-sa
    namespace: pgo
---
apiVersion: batch/v1
kind: Job
metadata:
  name: pgo-deploy
spec:
  backoffLimit: 0
  template:
    metadata:
      name: pgo-deploy
    spec:
      serviceAccountName: pgo-deployer-sa
      restartPolicy: Never
      containers:
        - name: pgo-deploy
          image: percona/percona-postgresql-operator:1.4.0-pgo-deployer
          imagePullPolicy: Always
          env:
            - name: DEPLOY_ACTION
              value: install
          volumeMounts:
            - name: deployer-conf
              mountPath: "/conf"
      volumes:
        - name: deployer-conf
          configMap:
            name: pgo-deployer-cm

Looks like the sed command fails because it does not find disable_auto_failover in this spec:

Once I added disable_fsgroup manually, we look to be installing as normal.

Now we are back to the backup error:

Wed Apr 5 20:26:45 UTC 2023 INFO: Image mode found: pgbackrest
Wed Apr 5 20:26:45 UTC 2023 INFO: Starting in ‘pgbackrest’ mode
time=“2023-04-05T20:26:45Z” level=info msg=“crunchy-pgbackrest starts”
time=“2023-04-05T20:26:45Z” level=info msg=“debug flag set to false”
time=“2023-04-05T20:26:45Z” level=info msg=“backrest backup command requested”
time=“2023-04-05T20:26:45Z” level=info msg=“backrest command will be executed for both local and s3 storage”
time=“2023-04-05T20:26:45Z” level=info msg=“command to execute is [pgbackrest backup --type=full --db-host=10.129.2.175 --db-path=/pgdata/dev-eng-pg-cluster ; pgbackrest backup --type=full --db-host=10.129.2.175 --db-path=/pgdata/dev-eng-pg-cluster --repo1-type=s3 --no-repo1-s3-verify-tls]”
time=“2023-04-05T20:29:34Z” level=info msg=“output=
time=“2023-04-05T20:29:34Z” level=info msg="stderr=[WARN: option ‘repo1-retention-full’ is not set for ‘repo1-retention-full-type=count’, the repository may run out of space\n HINT: to retain full backups indefinitely (without warning), set option ‘repo1-retention-full’ to the maximum.\nERROR: [055]: unable to load info file ‘/dev-eng/postgres-cluster/backup/db/backup.info’ or ‘/dev-eng/postgres-cluster/backup/db/backup.info.copy’:\n FileMissingError: unable to open missing file ‘/dev-eng/postgres-cluster/backup/db/backup.info’ for read\n FileMissingError: unable to open missing file ‘/dev-eng/postgres-cluster/backup/db/backup.info.copy’ for read\n HINT: backup.info cannot be opened and is required to perform a backup.\n HINT: has a stanza-create been performed?\nWARN: option ‘repo1-retention-full’ is not set for ‘repo1-retention-full-type=count’, the repository may run out of space\n HINT: to retain full backups indefinitely (without warning), set option 'repo1-retention-…
time=“2023-04-05T20:29:34Z” level=fatal msg=“command terminated with exit code 82”

It looks like the culprit is adding a value to repoPath. While have the value /dev-eng/postgres-cluster/ in repoPath instructed the backups to go the right s3 bucket prefix, my-bucket-name/dev-eng/postgres-cluster/ it causes the FileMissingError stated above.

Is this a known issue by chance? Is there a different way to specify the bucket prefix other than repoPath?

Thanks…