Description:
Running Percona Operator for PostgreSQL version 2.4.0 on OKD, I configured the backups to write on a Minio instance inside the cluster, but it seems pgbackrest doesn’t support HTTP connections.
Steps to Reproduce:
Here is the configuration for the PerconaPGCluster.spec.backups.pgbackrest.global CR and the configured repos:
...
global:
log-level-console: info
log-level-file: debug
repo1-path: /pgbackrest/postgres-operator/prod/repo1
repo1-retention-full: "3"
repo1-retention-full-type: count
repo1-s3-uri-style: path
repo1-storage-port: "9000"
repo1-storage-verify-tls: "n"
...
repos:
- name: repo1
s3:
bucket: pgdb-prod-backup
endpoint: myMinioHostname
region: us-east-1
schedules:
full: 0 0 * * *
...
Here is the resulting pgbackrest configuration inside the -instance- pod:
bash-4.4$ cat /etc/pgbackrest/conf.d/pgbackrest_instance.conf
# Generated by postgres-operator. DO NOT EDIT.
# Your changes will not be saved.
[global]
log-level-console= info
log-level-file = debug
log-path = /pgdata/pgbackrest/log
repo1-path = /pgbackrest/postgres-operator/prod/repo1
repo1-retention-full = 3
repo1-retention-full-type = count
repo1-s3-bucket = mybucket
repo1-s3-endpoint = myMinioHostname
repo1-s3-region = us-east-1
repo1-s3-uri-style = path
repo1-storage-port = 443
repo1-storage-verify-tls = n
repo1-type = s3
[db]
pg1-path = /pgdata/pg16
pg1-port = 5432
pg1-socket-path = /tmp/postgres
Version:
Percona Operator for PostgreSQL version 2.4.0
Logs:
Running a pgbackrest info inside the -instance- pod fails with TLS error:
bash-4.4$ pgbackrest info
stanza: [invalid]
status: error (other)
[ServiceError] TLS error [1:336130315] wrong version number
[ServiceError] on 2 retries from 120105-180207ms: TLS error [1:336130315] wrong version number
cipher: none
Expected Result:
The flag “repo1-storage-verify-tls” should either accept self-signed certificates or use an HTTP connection.
Here is a correct pgbackrest info output:
bash-4.4$ pgbackrest info
stanza: db
status: ok
cipher: none
db (current)
wal archive min/max (16): 000000030000000000000005/000000030000000000000007
full backup: 20241209-104928F
timestamp start/stop: 2024-12-09 10:49:28+00 / 2024-12-09 10:49:35+00
wal start/stop: 000000030000000000000006 / 000000030000000000000006
database size: 30.7MB, database backup size: 30.7MB
repo1: backup set size: 4.1MB, backup size: 4.1MB
Actual Result:
See above for the TLS error
Additional Information:
The original operator from Crunchydata supported this configuration and it seems it worked for some people S3 backups without TLS · Issue #3155 · CrunchyData/postgres-operator · GitHub, but thus far I couldn’t reproduce it, most likely they meant self-signed certificates and not HTTP connections.
I also found these issues, but I couldn’t find many details:
In the source code for pgbackrest, inside storage.c at line 1249 pgbackrest/src/storage/s3/storage.c at main · pgbackrest/pgbackrest · GitHub there doesn’t seem to be any options for not using TLS/using a normal TCP connection.
The documentation at https:// pgBackRest - Configuration Reference says:
This option provides the ability to enable/disable verification of the storage server TLS certificate
This configuration isn’t a problem per-se, it is actually correct to use TLS by default.
So far, my solution has been to use an OKD/OpenShift route with tls termination on Minio:
kind: Route
apiVersion: route.openshift.io/v1
metadata:
name: api
namespace: minio
labels:
component: minio
annotations:
openshift.io/host.generated: 'true'
spec:
host: api-minio.myhostname
to:
kind: Service
name: minio-api
weight: 100
port:
targetPort: 9000
tls:
termination: edge
wildcardPolicy: None
Then, I used port 443 to connect:
global:
...
repo1-storage-port: "443"
...
If you are using vanilla Kubernetes or other solutions, see https:// Network Encryption (TLS) — MinIO Object Storage for Linux on how to create self-signed certificates for your Minio.
I hope this helps someone else working on this configuration, it took me some troubleshooting to understand this was not the correct way to do things.