Postgres cluster to cluster replication with pg_tde encyption at DBs and WALs

Hello,

We have a full working primary cluster (A) with pg_tde encyption at DBs and WALs.

We want to replicate its data to a different standby (B) cluster , but still without any success. Fortunately we are on a Lab ,trying to have a working proof of concept . Any help is welcomed. Our cluster B /etc/patroni/postgresql.yml is the following

name             : pgb1
scope            : percona_b_scope
namespace        : /pg_cluster
etcd3 : { hosts  : 192.168.116.135:2379, 192.168.116.136:2379, 192.168.116.137:2379 }
restapi:
  listen         : 192.168.116.135:8008
  connect_address: 192.168.116.135:8008
postgresql:
  cluster_name   : percona_b_cluster
  listen         : 192.168.116.135:5432
  connect_address: 192.168.116.135:5432
  data_dir       : /var/lib/pgsql/18/data
  bin_dir        : /usr/pgsql-18/bin
  pgpass         : /tmp/pgpass
  parameters     : { unix_socket_directories: /run/postgresql }
  basebackup     : { checkpoint: fast }
  watchdog       : { safety_margin: 5, device: /dev/watchdog, mode: required  }  # mode values: off, automatic, required
  authentication :
    replication  : { username: replicator , password: SomePassword }
    superuser    : { username: postgres   , password: SomePassword }
    rewind       : { username: pgrewind   , password: SomePassword }
  create_replica_methods:
    - basebackup
bootstrap:
  dcs:
    standby_cluster:
      host                   : pga1.local.lan,pga2.local.lan,pga3.local.lan
      port                   : 5432
      primary_slot_name      : percona_b_slot
      create_replica_methods : [ basebackup ]
    ttl: 30
    loop_wait: 10
    retry_timeout: 10
    maximum_lag_on_failover: 1048576
    maximum_lag_on_syncnode: 15000000
    synchronous_mode: false
    postgresql:
      use_pg_rewind: true
      use_slots: true
      parameters:
        default_table_access_method: tde_heap
        pg_tde.wal_encrypt: 'on'
        shared_preload_libraries: pg_tde
        listen_addresses         : '*'
        ssl                      : true
        ssl_key_file             : /etc/patroni/ssl/server.key
        ssl_cert_file            : /etc/patroni/ssl/server.crt
        ssl_ca_file              : /etc/patroni/ssl/ca.crt
        ssl_min_protocol_version : TLSv1.3
        ssl_groups               : prime256v1
        max_connections          : 2000
        password_encryption      : scram-sha-256
        synchronous_standby_names: ''
        hot_standby              : on
        wal_level                : replica
        wal_keep_segments        : 10
        max_wal_senders          : 5
        max_replication_slots    : 10
        wal_buffers              : 64MB
        min_wal_size             : 1GB
        wal_log_hints            : 'on'
        archive_mode             : 'on'
        logging_collector        : 'on'
        log_rotation_size        : 100MB
        max_wal_size             : 2GB
        archive_timeout          : 600s
        archive_command          : pgbackrest --stanza=pg_ha archive-push /var/lib/pgsql/18/data/pg_wal/%f
        shared_buffers           : 2GB
        work_mem                 : 16MB
        maintenance_work_mem     : 2GB
        max_worker_processes     : 16
        effective_cache_size     : 64GB
        fsync                    : on
        temp_buffers             : 4MB
        checkpoint_completion_target: 0.9
      recovery_conf:
        recovery_target_timeline: latest
        restore_command: pgbackrest --config=/etc/pgbackrest.conf --stanza=pg_ha archive-get %f "%p"
  initdb:
    - auth-host : scram-sha-256
    - auth-local: scram-sha-256
    - auth-local: trust
    - encoding  : UTF8
    - data-checksums
  pg_hba:
    - local   all         all                           trust
    - local   replication all                           trust
    - host    all         all        127.0.0.1/32       scram-sha-256
    - host    all         all        ::1/128            scram-sha-256
    - host    replication all        ::1/128            scram-sha-256
    - host    replication all        127.0.0.1/32       scram-sha-256
    - host    replication replicator 127.0.0.1/32       scram-sha-256
    - host    replication replicator 192.168.116.131/32 scram-sha-256
    - host    replication replicator 192.168.116.129/32 scram-sha-256
    - host    replication replicator 192.168.116.128/32 scram-sha-256
    - host    replication replicator 192.168.116.135/32 scram-sha-256
    - host    replication replicator 192.168.116.136/32 scram-sha-256
    - host    replication replicator 192.168.116.137/32 scram-sha-256
    - host    replication replicator 172.168.196.253/32 scram-sha-256
    - host    replication replicator 172.168.196.254/32 scram-sha-256
    - hostssl all         all        samenet            scram-sha-256
users:
  admin:
    password: admin
    options:
      - createrole
      - createdb
slots:
  permanent_logical_slot_name:
    type       : logical
    database   : postgres
    plugin     : wal2json
tags:
  nofailover   : false
  noloadbalance: false
  clonefrom    : false
  nosync       : true

the A configuration is quite similar . thanks

Hi,

Archive and restore commands need special wrappers to decrypt/encrypt WAL when it is encrypted. See an example here: Limitations of pg_tde - Percona Transparent Data Encryption for PostgreSQL

Thank you for your response Andrew,

We changed archive_command and restore_command in the /etc/patroni/postgresql.yml with the encrypt/decrypt ones.

(We had already set these commands in the dynamic configuration via “patronictl -c /etc/patroni/patroni.yml edit-config”. But we now changed the /etc/patroni/postgresql.yml as well.)

We also added these lines on the configuration based on the link you pointed out.

bin_name:
  pg_basebackup: pg_tde_basebackup
  pg_rewind: pg_tde_rewind

Our configuration on the Standby cluster is now this

name                     : pgb1
scope                    : percona_b_scope
namespace                : /pg_cluster
etcd3                    : { hosts: 192.168.116.135:2379, 192.168.116.136:2379, 192.168.116.137:2379 }
restapi:
  listen                 : 192.168.116.135:8008
  connect_address        : 192.168.116.135:8008
postgresql:
  cluster_name           : percona_b_cluster
  listen                 : 192.168.116.135:5432
  connect_address        : 192.168.116.135:5432
  bin_dir                : /usr/pgsql-18/bin
  data_dir               : /var/lib/pgsql/18/data
  pgpass                 : /tmp/pgpass
  parameters             : { unix_socket_directories: /run/postgresql }
  basebackup             : { checkpoint: fast , max-rate: '100M' , wal-method: fetch }
  watchdog               : { safety_margin: 5, device: /dev/watchdog, mode: required  }  # mode values: off, automatic, required
  authentication:
    replication          : { username: replicator , password: SomePassword }
    superuser            : { username: postgres   , password: SomePassword }
    rewind               : { username: pgrewind   , password: SomePassword }
  create_replica_methods : [ basebackup ]
  bin_name               : { pg_basebackup: pg_tde_basebackup , pg_rewind: pg_tde_rewind }

bootstrap:
  dcs:
    standby_cluster:
      host                   : pga1.local.lan,pga2.local.lan,pga3.local.lan
      port                   : 5432
      primary_slot_name      : percona_b_slot
      create_replica_methods : [ basebackup ]
      bin_name               : { pg_basebackup: pg_tde_basebackup , pg_rewind: pg_tde_rewind }
      basebackup             : { checkpoint: fast , max-rate: '100M' , wal-method: fetch }
    ttl: 30
    loop_wait: 10
    retry_timeout: 10
    maximum_lag_on_failover: 1048576
    maximum_lag_on_syncnode: 15000000
    synchronous_mode: false
    postgresql:
      pg_hba:
        - local   all         all                           trust
        - local   replication all                           trust
        - host    all         all        127.0.0.1/32       scram-sha-256
        - host    all         all        ::1/128            scram-sha-256
        - host    replication all        ::1/128            scram-sha-256
        - host    replication all        127.0.0.1/32       scram-sha-256
        - host    replication replicator 127.0.0.1/32       scram-sha-256
        - host    replication replicator 192.168.116.131/32 scram-sha-256
        - host    replication replicator 192.168.116.129/32 scram-sha-256
        - host    replication replicator 192.168.116.128/32 scram-sha-256
        - host    replication replicator 192.168.116.135/32 scram-sha-256
        - host    replication replicator 192.168.116.136/32 scram-sha-256
        - host    replication replicator 192.168.116.137/32 scram-sha-256
        - host    replication replicator 172.168.196.253/32 scram-sha-256
        - host    replication replicator 172.168.196.254/32 scram-sha-256
        - hostssl all         all        samenet            scram-sha-256
      use_pg_rewind: true
      use_slots: true
      parameters:
        default_table_access_method: tde_heap
        pg_tde.wal_encrypt: 'on'
        shared_preload_libraries: pg_tde
        listen_addresses         : '*'
        ssl                      : true
        ssl_key_file             : /etc/patroni/ssl/server.key
        ssl_cert_file            : /etc/patroni/ssl/server.crt
        ssl_ca_file              : /etc/patroni/ssl/ca.crt
        ssl_min_protocol_version : TLSv1.3
        ssl_groups               : prime256v1
        max_connections          : 2000
        password_encryption      : scram-sha-256
        synchronous_standby_names: ''
        hot_standby              : on
        wal_level                : replica
        wal_keep_segments        : 10
        max_wal_senders          : 5
        max_replication_slots    : 10
        wal_buffers              : 64MB
        min_wal_size             : 1GB
        wal_log_hints            : 'on'
        archive_mode             : 'on'
        logging_collector        : 'on'
        log_rotation_size        : 100MB
        max_wal_size             : 2GB
        archive_timeout          : 600s
        archive_command          : '/usr/pgsql-18/bin/pg_tde_archive_decrypt %f %p "pgbackrest --config=/etc/pgbackrest.conf --stanza=pg_ha archive-push %%p"'
        shared_buffers           : 2GB
        work_mem                 : 16MB
        maintenance_work_mem     : 2GB
        max_worker_processes     : 16
        effective_cache_size     : 64GB
        fsync                    : on
        temp_buffers             : 4MB
        checkpoint_completion_target: 0.9
      recovery_conf:
        recovery_target_timeline: latest
        restore_command: "/usr/pgsql-18/bin/pg_tde_restore_encrypt %f %p \"pgbackrest --config=/etc/pgbackrest.conf --stanza=pg_ha archive-get %%f \\\"%%p\\\"\""
  initdb:
    - auth-host : scram-sha-256
    - auth-local: scram-sha-256
    - auth-local: trust
    - encoding  : UTF8
    - set       : shared_preload_libraries=pg_tde
    - data-checksums

users:
  admin:
    password: admin
    options:
      - createrole
      - createdb
slots:
  permanent_logical_slot_name:
    type       : logical
    database   : postgres
    plugin     : wal2json
tags:
  nofailover   : false
  noloadbalance: false
  clonefrom    : false
  nosync       : true

The Primary cluster is running normally with encryption on databases and wal. backup and restore with pgbackrest also works on the Primary cluster.

The problem is when we try to start the Standby cluster. Starting the patroni service on the Standby cluster leads to:

sudo -u postgres /usr/bin/patroni /etc/patroni/postgresql.yml
2026-03-05 16:58:55,215 INFO: Selected new etcd server http://192.168.116.136:2379
2026-03-05 16:58:55,226 WARNING: postgresql parameter listen_addresses=* failed validation, defaulting to None
2026-03-05 16:58:55,227 INFO: No PostgreSQL configuration items changed, nothing to reload.
2026-03-05 16:58:55,278 INFO: Lock owner: None; I am pgb1
2026-03-05 16:58:55,368 INFO: trying to bootstrap a new standby leader
pg_tde_basebackup: warning: the source has WAL keys, but no WAL encryption configured for the target backups
pg_tde_basebackup: detail: This may lead to exposed data and broken backup.
pg_tde_basebackup: hint: Run pg_basebackup with -E to encrypt streamed WAL.
2026-03-05 16:59:05,231 INFO: Lock owner: None; I am pgb1
2026-03-05 16:59:05,231 INFO: not healthy enough for leader race
2026-03-05 16:59:05,321 INFO: bootstrap_standby_leader in progress
2026-03-05 16:59:13,592 INFO: replica has been created using basebackup
2026-03-05 16:59:13,593 INFO: bootstrapped clone from remote member postgresql://pga1.local.lan,pga2.local.lan,pga3.local.lan:5432
2026-03-05 16:59:13,653 INFO: establishing a new patroni heartbeat connection to postgres
2026-03-05 16:59:13,744 INFO: establishing a new patroni heartbeat connection to postgres
2026-03-05 16:59:13,805 INFO: establishing a new patroni heartbeat connection to postgres
2026-03-05 16:59:13,929 INFO: postmaster pid=25242
2026-03-05 16:59:13.935 EET [25242] LOG:  registered custom resource manager "pg_tde" with ID 140
192.168.116.135:5432 - no response
2026-03-05 16:59:13.943 EET [25242] LOG:  tde_shmem_request: requested 71628840 bytes
2026-03-05 16:59:14,168 INFO: establishing a new patroni heartbeat connection to postgres
2026-03-05 16:59:14,327 INFO: establishing a new patroni heartbeat connection to postgres
2026-03-05 16:59:14.421 EET [25242] LOG:  creating DSA area of size 262144
2026-03-05 16:59:14.440 EET [25242] LOG:  redirecting log output to logging collector process
2026-03-05 16:59:14.440 EET [25242] HINT:  Future log output will appear in directory "log".
192.168.116.135:5432 - accepting connections
192.168.116.135:5432 - accepting connections
2026-03-05 16:59:15,043 INFO: establishing a new patroni heartbeat connection to postgres
2026-03-05 16:59:15,303 INFO: initialized a new cluster
2026-03-05 16:59:15,436 INFO: no action. I am (pgb1), the standby leader with the lock
2026-03-05 16:59:15,615 INFO: establishing a new patroni restapi connection to postgres
2026-03-05 16:59:25,397 INFO: no action. I am (pgb1), the standby leader with the lock
2026-03-05 16:59:35,350 INFO: no action. I am (pgb1), the standby leader with the lock

the output of list

patronictl -c /etc/patroni/postgresql.yml list
+ Cluster: percona_b_scope (7612590333590504236) -----------------+----+-------------+-----+------------+-----+--------------+
| Member | Host            | Role           | State               | TL | Receive LSN | Lag | Replay LSN | Lag | Tags         |
+--------+-----------------+----------------+---------------------+----+-------------+-----+------------+-----+--------------+
| pgb1   | 192.168.116.135 | Standby Leader | in archive recovery | 19 |             |     |            |     | nosync: true |
| pgb2   | 192.168.116.136 | Replica        | streaming           | 19 |  0/35000000 |   0 | 0/35000000 |   0 | nosync: true |
| pgb3   | 192.168.116.137 | Replica        | streaming           | 19 |  0/35000000 |   0 | 0/35000000 |   0 | nosync: true |
+--------+-----------------+----------------+---------------------+----+-------------+-----+------------+-----+--------------+