Hi,
I am testing a legacy backup system and am having problems at the prepare step.
The backup is to S3, of mysql 8.0.21, using xtrabackup version 8.0.25-17 based on MySQL server 8.0.25 Linux (x86_64) (revision id: d27028b)
The error I receive is:
# xtrabackup --prepare --apply-log-only --target-dir=test-restore/
xtrabackup: recognized client arguments: --prepare=1 --apply-log-only=1 --target-dir=test-restore/
xtrabackup version 8.0.25-17 based on MySQL server 8.0.25 Linux (x86_64) (revision id: d27028b)
xtrabackup: cd to /root/test-restore/
xtrabackup: This target seems to be not prepared yet.
Number of pools: 1
Operating system error number 2 in a file operation.
The error means the system cannot find the path specified.
xtrabackup: Warning: cannot open ./xtrabackup_logfile. will try to find.
Operating system error number 2 in a file operation.
The error means the system cannot find the path specified.
xtrabackup: Fatal error: cannot find ./xtrabackup_logfile.
The xtrabackup_logfile is indeed missing:
# ls -l test-restore/xtr*
-rw-r--r--. 1 root root 77 May 3 00:07 test-restore/xtrabackup_binlog_info.00000000000000000000
-rw-r--r--. 1 root root 36 May 3 00:07 test-restore/xtrabackup_binlog_info.00000000000000000001
-rw-r--r--. 1 root root 105 May 7 03:51 test-restore/xtrabackup_checkpoints
-rw-r--r--. 1 root root 161 May 3 00:07 test-restore/xtrabackup_checkpoints.00000000000000000000
-rw-r--r--. 1 root root 36 May 3 00:07 test-restore/xtrabackup_checkpoints.00000000000000000001
-rw-r--r--. 1 root root 734 May 7 03:51 test-restore/xtrabackup_info
-rw-r--r--. 1 root root 783 May 3 00:07 test-restore/xtrabackup_info.00000000000000000000
-rw-r--r--. 1 root root 29 May 3 00:07 test-restore/xtrabackup_info.00000000000000000001
-rw-r--r--. 1 root root 884276 May 3 00:07 test-restore/xtrabackup_logfile.00000000000000000000
-rw-r--r--. 1 root root 32 May 3 00:07 test-restore/xtrabackup_logfile.00000000000000000001
-rw-r--r--. 1 root root 95 May 3 00:07 test-restore/xtrabackup_tablespaces.00000000000000000000
-rw-r--r--. 1 root root 36 May 3 00:07 test-restore/xtrabackup_tablespaces.00000000000000000001
It’s missing from all attempts to make a full backup.
The command that creates this backup on S3 is:
xtrabackup --backup --no-lock --stream=xbstream --target-dir="${DAILY_BACKUP_DIR}" --extra-lsndir="${DAILY_BACKUP_DIR}" \
--keyring-file-data=${KEYRING_NAME} \
--host=${DB_HOST} --port=${DB_PORT} --user=${DB_USER} --password="${DB_PASS}" \
--parallel=${NUM_THREADS} | \
xbcloud put --storage=s3 \
--s3-endpoint="${S3_ENDPOINT}" \
--s3-access-key="${S3_ACCESSKEY}" \
--s3-secret-key="${S3_SECRETKEY}" \
--s3-bucket="${S3_BUCKET}" \
--parallel=${NUM_THREADS} \
"${HOSTNAME}/${BACKUP_NAME}/full"
This is blocking me from preparing a backup. Specifically I am trying to prepare a base (full) backup in order to try applying incremental backups to it. This is happening on the first, base prepare.
Thankyou in advance
Hello @toby5box
Can you check full backup log ? See if it’s completed successfully without any error.
I think you didn’t use xbcloud to download the backup from s3. xbcloud does this splitting of files.
So please use xbcloud get <> | xbstream and it will merge the files.
You have all the files but only in format that xbcloud and xbstream understands. Please follow the steps from the documentation : Use the xbcloud binary with Amazon S3 - Percona XtraBackup
1 Like
The backups always complete without error.
Here’s a snippet of the log:
May 8 00:06:50 db1-new CROND[3973240]: (root) CMDOUT (250508 00:06:50 xbcloud: successfully uploaded chunk: db1-new/Thu-2025-05-08-00-00-02/full/ib_buffer_pool.00000000000000000000, size: 10405383)
May 8 00:06:51 db1-new CROND[3973240]: (root) CMDOUT (xtrabackup: Transaction log of lsn (1643822304659) to (1643823311018) was copied.)
May 8 00:06:51 db1-new CROND[3973240]: (root) CMDOUT (2025-05-08T00:06:51.419302Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'keyring_file')
May 8 00:06:51 db1-new CROND[3973240]: (root) CMDOUT (2025-05-08T00:06:51.419375Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'daemon_keyring_proxy_plugin')
May 8 00:06:51 db1-new CROND[3973240]: (root) CMDOUT (250508 00:06:51 completed OK!)
May 8 00:06:52 db1-new CROND[3973240]: (root) CMDOUT (250508 00:06:52 xbcloud: Upload completed.)
May 8 00:06:52 db1-new CROND[3973240]: (root) CMDOUT (Completed 105 Bytes/105 Bytes (1.4 KiB/s) with 1 file(s) remaining#015upload: ../opt/x/backup-lns-data/Thu-2025-05-08-00-00-02/xtrabackup_checkpoints to s3://x-database-backups/db1-new/Thu-2025-05-08-00-00-02/full-lsn//opt/x/backup-lns-data/Thu-2025-05-08-00-00-02/xtrabackup_checkpoints)
May 8 00:06:53 db1-new CROND[3973240]: (root) CMDOUT (Completed 723 Bytes/723 Bytes (9.0 KiB/s) with 1 file(s) remaining#015upload: ../opt/x/backup-lns-data/Thu-2025-05-08-00-00-02/xtrabackup_info to s3://x-database-backups/db1-new/Thu-2025-05-08-00-00-02/full-lsn//opt/x/backup-lns-data/Thu-2025-05-08-00-00-02/xtrabackup_info)
May 8 00:06:53 db1-new CROND[3973240]: (root) CMDOUT (All backup operations complete. Total elapsed time: 0:06:51)
Satya
No, I’m using the file based backup, xbcloud, not xbstream.
I believe an xbstream format backup does not have this problem, but that’s a different topic. Before switching to xbstream I want to understand why I am having this problem with xbcloud in the legacy configuration and if it is a genuine bug.
(I’m also re-testing xbstream format right now as I need to be sure that we do in fact have at least one restorable backup. But I still would like to understand the problem with xbcloud.)
Indeed, the xbstream format when streamed to s3 as a single object does contain the xtrabackup_logfile:
...
performance_schema/binary_log_trans_1681.sdi
performance_schema/tls_channel_stat_1682.sdi
demo/db.opt
mysql-bin.037583
mysql-bin.index
xtrabackup_binlog_info
xtrabackup_logfile
xtrabackup_checkpoints
ib_buffer_pool
backup-my.cnf
xtrabackup_info
xtrabackup_tablespaces
This is the last few lines output of aws s3 cp s3://x-database-backups/db1-new/2025-05-08-16-02-38/full.xbstream - |xbstream -v -x -C test-restore
And in this case, the prepare step completes successfully and I can start mysqld on the results.
1 Like
@toby5box if you xbcloud, it will still use xbstream. observe --stream=xbstream parameter to xtrabackup. xbstream is just a format for streaming.
You have used xtrabackup --backup |xbcloud put... To use the backup, you HAVE to use xbcloud again. xbcloud get | xbstream -x -C restore --parallel=8. After this, you can do prepare.
1 Like
Thanks. I have tested this and it did work:
xbcloud get --storage=s3 --s3-bucket="${S3_BUCKET}" --parallel=${NUM_THREADS} $S3PATH \
| xbstream -x -C $TARGET_DIR --parallel=$NUM_THREADS
xtrabackup --prepare --keyring-file-data=`pwd`/mysql.keyring --no-apply-log --target-dir=$TARGET_DIR
Is there any reason to not to store the xbstream as a single object on S3 versus the xbcloud many-objects backup? It seems that the single object format (capturing the output of xtrabackup directly) is more efficient to download.
My logic was poor. It seems that the combination of multi-objects (xbcloud) and multithreading is likely to be faster to transfer. Even as much as 2x in my tests.