Xtrabackup: error: failed to backup rocksdb WAL files

MySQL & MariaDB Percona XtraBackup
1

I specified the --rocksdb-wal-dir parameter, but the program seems to follow the value of performance_schema.log_status .

"RocksDB":
{
    "wal_files":
    [
        {
            "path_name": "/archive/000003.log",
            "log_number": 3,
            "size_file_bytes": 93
        },
        {
            "path_name": "/archive/000011.log",
            "log_number": 11,
            "size_file_bytes": 19
        },

xtrabackup: recognized server arguments: --rocksdb_wal_dir=/apps/dbdat/percona8_data3306/wal_log 
xtrabackup: recognized client arguments: --backup=1 --target-dir=/apps/mysql_backup/241 --user=xb --password=* --socket=/tmp/mysql3306.sock --parallel=8 
./xtrabackup version 8.0.22-15 based on MySQL server 8.0.22 Linux (x86_64) (revision id: fea8a0e)
210813 18:06:33  version_check Connecting to MySQL server with DSN 'dbi:mysql:;mysql_read_default_group=xtrabackup;mysql_socket=/tmp/mysql3306.sock' as 'xb'  (using password: YES
).
210813 18:06:33  version_check Connected to MySQL server
210813 18:06:33  version_check Executing a version check against the server...
210813 18:06:33  version_check Done.
210813 18:06:33 Connecting to MySQL server host: localhost, user: xb, password: set, port: not set, socket: /tmp/mysql3306.sock
Using server version 8.0.22-13
xtrabackup: uses posix_fadvise().
..................
..................
210813 17:36:13 Finished backing up non-InnoDB tables and files
210813 17:36:13 xtrabackup: Creating RocksDB checkpoint
210813 17:36:13 Executing FLUSH NO_WRITE_TO_BINLOG BINARY LOGS
210813 17:36:13 Selecting LSN and binary log position from p_s.log_status
210813 17:36:13 [00] Copying /apps/dbdat/percona8_data3306/log/mysql-bin.002074 to /apps/mysql_backup/241/mysql-bin.002074 up to position 196
210813 17:36:13 [00]        ...done
210813 17:36:13 [00] Writing /apps/mysql_backup/241/mysql-bin.index
210813 17:36:13 [00]        ...done
xtrabackup: xtrabackup: File 'archive/003566.log' not found (OS errno 2 - No such file or directory)xtrabackup: xtrabackup: xtrabackup: File 'archive/005661.log' not found (OS err
no 2 - No such file or directory)

File 'archive/003021.log' not found (OS errno 2 - No such file or directory)210813 17:36:13 [08] Copying /apps/dbdat/percona8_data3306/wal_log/006209.log to /apps/mysql_backup/241
/.rocksdb/006209.log up to position 77366546
Operating system error number 2 in a file operation.

*************************** 1. row ***************************
Grants for xb@localhost: GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, RELOAD, SHUTDOWN, PROCESS, FILE, REFERENCES, INDEX, ALTER, SHOW DATABASES, SUPER, CREATE TEMPORARY TABLES, LOCK TABLES, EXECUTE, REPLICATION SLAVE, REPLICATION CLIENT, CREATE VIEW, SHOW VIEW, CREATE ROUTINE, ALTER ROUTINE, CREATE USER, EVENT, TRIGGER, CREATE TABLESPACE, CREATE ROLE, DROP ROLE ON *.* TO `xb`@`localhost`
*************************** 2. row ***************************
Grants for xb@localhost: GRANT APPLICATION_PASSWORD_ADMIN,AUDIT_ADMIN,BACKUP_ADMIN,BINLOG_ADMIN,BINLOG_ENCRYPTION_ADMIN,CLONE_ADMIN,CONNECTION_ADMIN,ENCRYPTION_KEY_ADMIN,GROUP_REPLICATION_ADMIN,INNODB_REDO_LOG_ARCHIVE,INNODB_REDO_LOG_ENABLE,PERSIST_RO_VARIABLES_ADMIN,REPLICATION_APPLIER,REPLICATION_SLAVE_ADMIN,RESOURCE_GROUP_ADMIN,RESOURCE_GROUP_USER,ROLE_ADMIN,SERVICE_CONNECTION_ADMIN,SESSION_VARIABLES_ADMIN,SET_USER_ID,SHOW_ROUTINE,SYSTEM_USER,SYSTEM_VARIABLES_ADMIN,TABLE_ENCRYPTION_ADMIN,XA_RECOVER_ADMIN ON *.* TO `xb`@`localhost`
2 Likes
2

Hi @wonderful1984 .

Can you please paste your my.cnf, an ls of /archive folder and an ls of /apps/dbdat/percona8_data3306/wal_log as well a full copy of your backup log?

Thanks

1 Like
3

percona8_3306_myrocks.txt (11.7 KB)

xtrabackup.log (474.8 KB)

]$ ls -lh / |grep apps
drwxr-xr-x   15 apps apps  245 Aug 13 10:03 apps

]$ ls -lh /apps/|grep mysql_backup
drwxrwxr-x  2 apps apps   37 Aug 13 18:08 mysql_backup

]$ ls -lh /apps/mysql_backup/
total 476K
-rw-rw-r-- 1 apps apps 475K Aug 13 18:06 xtrabackup.log

~]$ ls -lh /apps/dbdat/percona8_data3306/wal_log|head
total 21G
-rw-r----- 1 apps apps 246M Aug 13 18:08 006270.log
-rw-r----- 1 apps apps 242M Aug 13 18:10 006273.log
-rw-r----- 1 apps apps 246M Aug 13 18:11 006275.log
-rw-r----- 1 apps apps 244M Aug 13 18:13 006277.log
-rw-r----- 1 apps apps 240M Aug 13 18:15 006283.log
-rw-r----- 1 apps apps 240M Aug 13 18:17 006285.log
-rw-r----- 1 apps apps 246M Aug 13 18:18 006287.log
-rw-r----- 1 apps apps 242M Aug 13 18:20 006289.log
-rw-r----- 1 apps apps 238M Aug 13 18:22 006291.log

~]$ ls -lh /apps/dbdat/percona8_data3306/wal_log/archive/|head
total 417G
-rw-r----- 1 apps apps   93 Aug 12 12:18 000003.log
-rw-r----- 1 apps apps   19 Aug 12 12:18 000011.log
-rw-r----- 1 apps apps   93 Aug 12 12:28 000015.log
-rw-r----- 1 apps apps   19 Aug 12 12:28 000022.log
-rw-r----- 1 apps apps   93 Aug 12 17:19 000026.log
-rw-r----- 1 apps apps   19 Aug 12 17:19 000033.log
-rw-r----- 1 apps apps 188M Aug 12 19:39 000037.log
-rw-r----- 1 apps apps 193M Aug 12 19:40 000044.log
-rw-r----- 1 apps apps 188M Aug 12 19:41 000046.log
1 Like
4

Hi, @Marcelo_Altmann Is there any information update? Thanks.

1 Like
5

Hi @wonderful1984 .

This is a bug on Xtrabackup. I’ve created [PXB-2585] xtrabackup fail to backup archive rockdb wal files - Percona JIRA, you can subscribe to it in order to get updates on its progress.

1 Like
6

workaround：
Step 1: ln -s /apps/dbdat/percona8_data3306/wal_log/archive /apps/dbdat/percona8_data3306/archive

------- I find that the program does not create this directory. An empty directory will not work, You need to create this directory
Step 2: mkdir -p /apps/mysql_backup/${backup_path}/.rocksdb/archive/

1 Like
7

You will also have problems at prepare of incremental backups and copy-back.
[PXB-2585] xtrabackup fail to backup archive rockdb wal files - Percona JIRA is fixed and will be part of PXB 8.0.26 release.

1 Like