Hey,
wondering why you added the requirement to decrypt/encrypt the WAL if pgbackrest is used in the limitations documentation.
I remember it wasn’t there before.
We are currently using pgbackrest without encrypt/decrypt and it works fine. No problems with Restores or anything else. Is it just to make sure that the WAL Files could be used on a (non encrypted) standby or other external tool?
The limitation:- Limitations of pg_tde - Percona Transparent Data Encryption for PostgreSQL mentions that the following options should be set in PgBackRest to allow pg_tde WAL encryption.
Need to Configure/Enable:
pg_tde_archive_decrypt
`pg_tde_restore_encrypt
Need to disable them:
archive-async
archive-header-check
checksum-page
PgBackRest will still work if you are not using the WAL encryption option; however, if you want to apply it, then it must be configured as mentioned above. The underlying tool, PgBackRest itself, doesn’t need to know about the encryption or decryption mechanism. The tool will simply recognise the WALs once they are decrypted by the pg_tde process.
E.g,
archive_command='pg_tde_archive_decrypt %f %p "pgbackrest --stanza=your_stanza archive-push %%p"'
Yes, other external tools can use the WAL unless they are managed by the pg_tde wal encryption.
That is why I am confused. We have WAL encryption activated but have not set the encrypt/decrypt commands and everything works just fine.
We have disabled the flags like archive-header-check.
Hi,
Offhand, it sounds like there’s a bit of communication confusion. The issue about tde is pgbackrest being able to understand the WAL when it’s encrypted. You may not have a problem by virtue of the manner your implementation.
If you think it’s worth your time, maybe you can describe the exact steps you are performing.
Hi Robert,
sure: we setup the postgres cluster on debian following the installation in your docs:
Percona Server for PostgreSQL 18.6.1
KMS is vault.
Afterwards we activated WAL encryption. To confirm that WAL encryption is working we inserted some data and used hexdump to verify it is encrypted.
This is the postgresql.conf we use:
shared_preload_libraries = 'pg_stat_statements, pg_tde'
pg_tde.wal_encrypt = on
archive_mode = on
archive_command = 'pgbackrest --stanza=main archive-push %p'
notice that no decrypt is used.
This is the pgbackrest.conf:
[global]
repo1-type=s3
repo1-path=/main
repo1-s3-uri-style=path
repo1-s3-endpoint=xxx.internal
repo1-s3-region=us-east-1
repo1-s3-bucket=dev-dbbackup
repo1-s3-key=xxx
repo1-s3-key-secret=xxx
repo1-storage-verify-tls=n
start-fast=y
process-max=4
archive-async=n
archive-header-check=n
[main]
pg1-path=/var/lib/postgresql/18/main
pg1-port=5432
pg1-user=postgres
To restore, we use no additional options. For PITR this works:
sudo -u postgres pgbackrest --stanza=main --type=time --target="2026-09-10 10:44:00+02" restore
For a complete restore:
sudo -u postgres pgbackrest --stanza=main restore
Either removing all postgres data with sudo rm -rf /var/lib/postgresql/18/main/ or using the --delta option works.
During restore I do not modify the restore_command of pgbackrest, so it is the default restore_command = 'pgbackrest --stanza=main archive-get %f "%p"'
pg_tde_archive_decrypt and pg_tde_archive_encrypt are not used at all.
My questions is why the documentation tells you that you must use these commands.
This notice was not there when we first set up the system a few months ago, so something had to happen in the meantime that this is “forced” now.
Hi @grill,
Your setup worked for a specific reason, and it has some edge cases.
With archive_command = 'pgbackrest ... archive-push %p', pgBackRest copies the WAL segments into the repo still encrypted. You’ve turned off archive-header-check and checksum-page, so pgBackRest never tries to read what’s inside them. It treats them as opaque files.
On restore, archive-get puts the same encrypted segments back into pg_wal. The restored server then decrypts them using the WAL key file in $PGDATA/pg_tde/, which came back with the backup and your principal key in Vault.
As long as the restored key file has a key for every LSN range you replay, recovery works. That’s what you’re seeing.
The restored WAL key file is a snapshot taken at backup time. The archive, though, keeps receiving WAL that may have been encrypted with keys created after that backup. For example, a new WAL key is generated when the server restarts, or when WAL encryption is toggled. If you do PITR from backup N to a point after one of those events, the restored cluster may not have the key for those segments, and recovery fails.
The same coupling shows up in a few other places:
pgbackrest verify or other tooling.The wrappers remove that coupling:
pg_tde_archive_decrypt stores plain WAL in the repo.pg_tde_restore_encrypt re-encrypts the WAL with the restoring cluster’s current key.The archive no longer depends on any particular snapshot of pg_tde/, so this is the configuration we test and support. Because the WAL in the repo is then unencrypted at the pg_tde level, please turn on pgBackRest repository encryption (repo1-cipher-type=aes-256-cbc plus repo1-cipher-pass) so the archive is still encrypted at rest.
As WAL encryption matured, the limitations page was updated to reflect what we actually test. It isn’t a sign that your current restores are wrong. It marks the line between what we’ve validated and a setup that works in the common case.
You can try to check on yourself:
SELECT pg_switch_wal();.--type=time to a point after the restart.If this succeeds in your environment, please share the result. We’d like to know.
For production, we recommend switching to:
archive_command = 'pg_tde_archive_decrypt %f %p "pgbackrest --stanza=main archive-push %%p"'
restore_command = 'pg_tde_restore_encrypt %f %p "pgbackrest --stanza=main archive-get %%f \"%%p\""'
Warm regards,
N.
Unanswered | Unsolved | Solved
MySQL, InnoDB, MariaDB and MongoDB are trademarks of their respective owners.
Copyright © 2006 - 2024 Percona LLC. All rights reserved.