Documenting my tests in using HTTP with pgbackrest and Minio

PostgreSQL Percona Operator for PostgreSQL
1

Description:

Running Percona Operator for PostgreSQL version 2.4.0 on OKD, I configured the backups to write on a Minio instance inside the cluster, but it seems pgbackrest doesn’t support HTTP connections.

Steps to Reproduce:

Here is the configuration for the PerconaPGCluster.spec.backups.pgbackrest.global CR and the configured repos:

...
      global:
        log-level-console: info
        log-level-file: debug
        repo1-path: /pgbackrest/postgres-operator/prod/repo1
        repo1-retention-full: "3"
        repo1-retention-full-type: count
        repo1-s3-uri-style: path
        repo1-storage-port: "9000"
        repo1-storage-verify-tls: "n"
...
      repos:
      - name: repo1
        s3:
          bucket: pgdb-prod-backup
          endpoint: myMinioHostname
          region: us-east-1
        schedules:
          full: 0 0 * * *
...

Here is the resulting pgbackrest configuration inside the -instance- pod:

bash-4.4$ cat /etc/pgbackrest/conf.d/pgbackrest_instance.conf
# Generated by postgres-operator. DO NOT EDIT.
# Your changes will not be saved.

[global]
log-level-console= info
log-level-file = debug
log-path = /pgdata/pgbackrest/log
repo1-path = /pgbackrest/postgres-operator/prod/repo1
repo1-retention-full = 3
repo1-retention-full-type = count
repo1-s3-bucket = mybucket
repo1-s3-endpoint = myMinioHostname
repo1-s3-region = us-east-1
repo1-s3-uri-style = path
repo1-storage-port = 443
repo1-storage-verify-tls = n
repo1-type = s3

[db]
pg1-path = /pgdata/pg16
pg1-port = 5432
pg1-socket-path = /tmp/postgres

Version:

Percona Operator for PostgreSQL version 2.4.0

Logs:

Running a pgbackrest info inside the -instance- pod fails with TLS error:

bash-4.4$ pgbackrest info

stanza: [invalid]
    status: error (other)
      [ServiceError] TLS error [1:336130315] wrong version number
                [ServiceError] on 2 retries from 120105-180207ms: TLS error [1:336130315] wrong version number
    cipher: none

Expected Result:

The flag “repo1-storage-verify-tls” should either accept self-signed certificates or use an HTTP connection.
Here is a correct pgbackrest info output:

bash-4.4$ pgbackrest info
stanza: db
    status: ok
    cipher: none

    db (current)
        wal archive min/max (16): 000000030000000000000005/000000030000000000000007

        full backup: 20241209-104928F
            timestamp start/stop: 2024-12-09 10:49:28+00 / 2024-12-09 10:49:35+00
            wal start/stop: 000000030000000000000006 / 000000030000000000000006
            database size: 30.7MB, database backup size: 30.7MB
            repo1: backup set size: 4.1MB, backup size: 4.1MB

Actual Result:

See above for the TLS error

Additional Information:

The original operator from Crunchydata supported this configuration and it seems it worked for some people S3 backups without TLS · Issue #3155 · CrunchyData/postgres-operator · GitHub, but thus far I couldn’t reproduce it, most likely they meant self-signed certificates and not HTTP connections.

I also found these issues, but I couldn’t find many details:

In the source code for pgbackrest, inside storage.c at line 1249 pgbackrest/src/storage/s3/storage.c at main · pgbackrest/pgbackrest · GitHub there doesn’t seem to be any options for not using TLS/using a normal TCP connection.

The documentation at https:// pgBackRest - Configuration Reference says:

This option provides the ability to enable/disable verification of the storage server TLS certificate

This configuration isn’t a problem per-se, it is actually correct to use TLS by default.

So far, my solution has been to use an OKD/OpenShift route with tls termination on Minio:

kind: Route
apiVersion: route.openshift.io/v1
metadata:
  name: api
  namespace: minio
  labels:
    component: minio
  annotations:
    openshift.io/host.generated: 'true'
spec:
  host: api-minio.myhostname
  to:
    kind: Service
    name: minio-api
    weight: 100
  port:
    targetPort: 9000
  tls:
    termination: edge
  wildcardPolicy: None

Then, I used port 443 to connect:

      global:
        ...
        repo1-storage-port: "443"
        ...

If you are using vanilla Kubernetes or other solutions, see https:// Network Encryption (TLS) — MinIO Object Storage for Linux on how to create self-signed certificates for your Minio.

I hope this helps someone else working on this configuration, it took me some troubleshooting to understand this was not the correct way to do things.

1 Like
2

See also Issues with configuring backup against S3 storage for more information

3

This is awesome, @SuperHammer ! Thank you for documenting it.

Let me talk to our PG team to verify pgbackrest capability to use HTTP for S3.

1 Like