Hello folks.
I wanna try zstd compression in my infrastructure. I took backup, everything was ok. But when I try to restore data from this backup - it was failed on decompress stage.
MySQL and xtrabackup versions:
~ mysql --version
mysql Ver 8.0.32-24 for Linux on x86_64 (Percona Server (GPL), Release ‘24’, Revision ‘e5c6e9d2’)
~ xtrabackup --version
2023-07-12T14:07:42.163137-00:00 0 [Note] [MY-011825] [Xtrabackup] recognized server arguments: --datadir=/data/mysql --tmpdir=/data/mysqltmp --innodb_flush_log_at_trx_commit=2 --innodb_file_per_table=1 --innodb_read_io_threads=16 --innodb_write_io_threads=16 --innodb_flush_method=O_DIRECT --log_bin=/data/mysql/mysql-bin --server-id=17221
xtrabackup version 8.0.32-26 based on MySQL server 8.0.32 Linux (x86_64) (revision id: 34cf2908)
Here is my workflow:
backup server:
~ xtrabackup --backup --compress=zstd --parallel=16 --compress-threads=8 --stream=xbstream --target_dir=/data/backup 2> /data/backup/2023-07-12_00-00-03_Wednesday.log | pv -s 5559283712 -p -f 2> /tmp/backup_progress | /usr/local/bin/aws s3 cp - s3://backups/2023-07-12_00-00-03_Wednesday.xbstream --sse --expected-size 5559283712
On new server:
~ /usr/local/bin/aws s3 cp s3://backups/2023-07-12_00-00-03_Wednesday.xbstream - | xbstream -x --parallel 10 -C /data/mysql
~ xtrabackup --decompress --remove-original --target-dir=/data/mysql > /data/mysql/decompress.log 2>&1
And on this stage I have an error:
2023-07-12T14:04:47.745000-00:00 0 [Note] [MY-011825] [Xtrabackup] recognized server arguments: --datadir=/data/mysql --tmpdir=/data/mysqltmp --innodb_flush_log_at_trx_commit=2 --innodb_file_per_table=1 --innodb_read_io_threads=16 --innodb_write_io_threads=16 --innodb_flush_method=O_DIRECT --log_bin=/data/mysql/mysql-bin --server-id=17221
2023-07-12T14:04:47.745265-00:00 0 [Note] [MY-011825] [Xtrabackup] recognized client arguments: --port=3306 --socket=/var/run/mysqld/mysqld.sock --compress --compress-threads=4 --parallel=4 --user=backup --password=* --decompress=1 --remove-original=1 --target-dir=/data/mysql
xtrabackup version 8.0.32-26 based on MySQL server 8.0.32 Linux (x86_64) (revision id: 34cf2908)
2023-07-12T14:04:47.745996-00:00 0 [Note] [MY-011825] [Xtrabackup] decompressing ./mysql.ibd.zst
2023-07-12T14:04:47.746185-00:00 0 [Note] [MY-011825] [Xtrabackup] decompressing ./xtrabackup_logfile.zst
2023-07-12T14:04:47.746385-00:00 0 [Note] [MY-011825] [Xtrabackup] decompressing ./undo_001.zst
2023-07-12T14:04:47.746532-00:00 0 [Note] [MY-011825] [Xtrabackup] decompressing ./test/start_recrawl_24.ibd.zst
sh: 1: zstd: not found
sh: 1: zstd: not found
cat: write error: Broken pipe
cat: write error: Broken pipe
sh: 1: zstd: not found
sh: 1: zstd: not found
cat: write error: Broken pipe
2023-07-12T14:04:47.847908-00:00 0 [ERROR] [MY-011825] [Xtrabackup] decrypt and decompress thread 0 failed.
2023-07-12T14:04:47.847984-00:00 0 [ERROR] [MY-011825] [Xtrabackup] decrypt and decompress thread 1 failed.
2023-07-12T14:04:47.848001-00:00 0 [ERROR] [MY-011825] [Xtrabackup] decrypt and decompress thread 2 failed.
2023-07-12T14:04:47.848034-00:00 0 [ERROR] [MY-011825] [Xtrabackup] decrypt and decompress thread 3 failed.
I tried less complicated implementation without streaming component:
xtrabackup --backup --compress=zstd --parallel=16 --compress-threads=8 --target_dir=/data/backup
scp to new server and decompress:
xtrabackup --decompress --remove-original --target-dir=/data/mysql > /data/mysql/decompress.log 2>&1
Result was the same. Any idea what I am doing wrong?