Appdynamics mysql backup error

MySQL & MariaDB Percona XtraBackup
1

I’m trying to do a full backup on AppDynamics contoller mysql db and I get the below error

xtrabackup --user=mysqluser --password=mysqlpwd --backup --datadir=<Controller_dir>/db/data/mysql --socket=./mysql.sock

xtrabackup: recognized server arguments: --datadir=/var/lib/mysql --datadir=<Controller_dir>/db/data/mysql
xtrabackup: recognized client arguments: --user=mysqluser --password=* --backup=1 --socket=./mysql.sock
231103 14:45:25 version_check Connecting to MySQL server with DSN ‘dbi:mysql:;mysql_read_default_group=xtrabackup;mysql_socket=./mysql.sock’ as ‘mysqluser’ (using password: YES).
231103 14:45:25 version_check Connected to MySQL server
231103 14:45:25 version_check Executing a version check against the server…
231103 14:45:28 version_check Done.
231103 14:45:28 Connecting to MySQL server host: localhost, user: mysqluser, password: set, port: not set, socket: ./mysql.sock
Using server version 5.7.39-enterprise-commercial-advanced-log
Warning: option ‘datadir’ has different values:
‘<Controller_dir>/db/data/mysql’ in defaults file
‘<Controller_dir>/db/data/’ in SHOW VARIABLES
xtrabackup version 2.4.28 based on MySQL server 5.7.40 Linux (x86_64) (revision id: 44a8f7b)
xtrabackup: uses posix_fadvise().
xtrabackup: cd to /opt/appdyn/controller/db/data/mysql
xtrabackup: open files limit requested 0, set to 1024
xtrabackup: using the following InnoDB configuration:
xtrabackup: innodb_data_home_dir = .
xtrabackup: innodb_data_file_path = ibdata1:12M:autoextend
xtrabackup: innodb_log_group_home_dir = ./
xtrabackup: innodb_log_files_in_group = 2
xtrabackup: innodb_log_file_size = 67108864
InnoDB: Number of pools: 1
InnoDB: Operating system error number 2 in a file operation.
InnoDB: The error means the system cannot find the path specified.
InnoDB: File ./ib_logfile0: ‘open’ returned OS error 71. Cannot continue operation
InnoDB: Cannot continue operation.

1 Like
2

Make sure you are running PXB as either root OS user, or a user that can read the MySQL data files. Also, check that SELinux/AppArmour have not re-enabled if previously disabled.

3

I’m running the pxb as root…!!

But I see the SELinux is enabled. can you share some documentation on why this needs to be disabled while we run this backup

4

SELinux is preventing access to the MySQL data files from any other program. You can set SELinux into permissive mode (does not block, but still logs activity). There are many HOWTOs on the web for creating/modifying SELinux rules. You’ll need to do this to allow PXB to run with SELinux enabled.

5

can you point me to documentation or suggest the SELinux rules to allow Percona to read mysql

Also, While search more on the backup commands I see few posts using the ‘innobackupex’ command, is it for older versions of Xtrabackup?

6

Yes. innobackupex is extremely old software.

7

I still get the same error:

xtrabackup --defaults-file=/opt/appdata/platform/controller/db/db.cnf --user=userid --password=pwd --backup --datadir=/opt/appdata/platform/controller/db/data/mysql --socket=./mysql.sock

xtrabackup: recognized server arguments: --datadir=/opt/appdata/platform/controller/db/data --tmpdir=/opt/appdata/platform/controller/db/data --innodb_open_files=300 --open_files_limit=40960 --innodb_flush_log_at_trx_commit=0 --innodb_io_capacity=200 --innodb_read_io_threads=8 --innodb_write_io_threads=8 --innodb_buffer_pool_size=64M --innodb_log_file_size=64M --innodb_log_buffer_size=2M --innodb_file_per_table=1 --datadir=/opt/appdata/platform/controller/db/data/mysql
xtrabackup: recognized client arguments: --user=root --password=* --backup=1 --socket=./mysql.sock
Can’t locate English.pm in @INC (you may need to install the English module) (@INC contains: /usr/local/lib64/perl5/5.32 /usr/local/share/perl5/5.32 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5) at - line 3.
BEGIN failed–compilation aborted at - line 3.
231109 12:27:50 Connecting to MySQL server host: localhost, user: root, password: set, port: not set, socket: ./mysql.sock
Using server version 5.7.41-enterprise-commercial-advanced-log
Warning: option ‘datadir’ has different values:
‘/opt/appdata/platform/controller/db/data/mysql’ in defaults file
‘/opt/appdata/platform/controller/db/data/’ in SHOW VARIABLES
xtrabackup version 2.4.28 based on MySQL server 5.7.40 Linux (x86_64) (revision id: 44a8f7b)
xtrabackup: uses posix_fadvise().
xtrabackup: cd to /opt/appdata/platform/controller/db/data/mysql
xtrabackup: open files limit requested 40960, set to 40960
xtrabackup: using the following InnoDB configuration:
xtrabackup: innodb_data_home_dir = .
xtrabackup: innodb_data_file_path = ibdata1:12M:autoextend
xtrabackup: innodb_log_group_home_dir = ./
xtrabackup: innodb_log_files_in_group = 2
xtrabackup: innodb_log_file_size = 67108864
InnoDB: Number of pools: 1
InnoDB: Operating system error number 2 in a file operation.
InnoDB: The error means the system cannot find the path specified.
InnoDB: File ./ib_logfile0: ‘open’ returned OS error 71. Cannot continue operation
InnoDB: Cannot continue operation.

8

OS Error 2 is “No such file or directory”
OS Error 71 is actually a mistype. https://bugs.mysql.com/bug.php?id=85016 and 71 also means “File not found.”

You need to look at SELinux audit logs and see if it is actively denying the open command.

I would also fix that issue as well.

Set SELinux in permissive mode and then run PXB. If it works, then you know SELinux is the problem.

9

This is also something to check. Looks like you are using --datadir=/opt/appdata/platform/controller/db/data/mysql in the parameter to xtrabackup, but your server datadir is one directory below as displayed in the SHOW VARIABLES warning.

Can you please try to backup using --datadir=/opt/appdata/platform/controller/db/data ?

10

I managed to create the backup using --backup & --compress option. A directory ‘xtrabackup_backupfiles’ is now created with compressed files.

As the next step I tried to do a --prepare from the db directory (where the --backup command was run) and get the below error.

xtrabackup --prepare
xtrabackup: recognized server arguments:
xtrabackup: recognized client arguments: --prepare=1
xtrabackup version 2.4.28 based on MySQL server 5.7.40 Linux (x86_64) (revision id: 44a8f7b)
xtrabackup: cd to /opt/appdata/platform/controller/db/xtrabackup_backupfiles/
xtrabackup: This target seems to be not prepared yet.
InnoDB: Number of pools: 1
InnoDB: Operating system error number 2 in a file operation.
InnoDB: The error means the system cannot find the path specified.
xtrabackup: Warning: cannot open ./xtrabackup_logfile. will try to find.
InnoDB: Operating system error number 2 in a file operation.
InnoDB: The error means the system cannot find the path specified.
xtrabackup: Fatal error: cannot find ./xtrabackup_logfile.
xtrabackup: Error: xtrabackup_init_temp_log() failed.

I do see the file ‘xtrabackup_logfile.qp’ in the ‘xtrabackup_backupfiles’ directory

I did try doing a decompress before running prepare but I get the below error though qpress is already installed.

xtrabackup --decompress
xtrabackup: recognized server arguments:
xtrabackup: recognized client arguments: --decompress=1
xtrabackup version 2.4.28 based on MySQL server 5.7.40 Linux (x86_64) (revision id: 44a8f7b)
231113 19:31:49 [01] decompressing ./sys/x@0024schema_table_statistics.frm.qp
sh: qpress: command not found
cat: write error: Broken pipe
Error: decrypt and decompress thread 0 failed.

11

Yes, you must decompress before you can compare. How did you verify qpress is installed? Use which qpress to verify.

12

Looks like we dont have qpress installed…!!
Why is not listed in the dependencies if percona is using qpress for compression?

13

That is an excellent question! :slight_smile:
Here’s the JIRA I opened on that very issue over 3 years ago.
https://jira.percona.com/browse/PXB-2134
But now, in PXB8, qpress is no longer used and zstd is the default. Once you get to MySQL 8, you shouldn’t have any issues.

14

Where can we download the latest version of qpress? we have 11.1 and it is flagged in our vulnerability scan.

15 
$ qpress -v
qpress 1.1 - Copyright 2006-2010 Lasse Reinhold - www.quicklz.com
Using QuickLZ 1.4.1 compression library

https://repo.percona.com/tools/yum/release/9/RPMS/x86_64/qpress-11-3.el9.x86_64.rpm
https://repo.percona.com/tools/apt/pool/main/q/qpress/

16

is there a quick way to findout which version of qpress is installed? if I do a qpress -v, both the versions show they are qpress 1.1 using QuickLZ 1.4.1

17

Looks like you are on the latest version. When you upgrade to Xtrabackup 8, qpress was removed and replaced with ZSTD