PXB 8.4.0 causes Percona Server 8.4.2-2 to crash when a backup is run

MySQL & MariaDB Percona XtraBackup
1

When I try to run a simple backup on a Percona Server 8.4.2-2 using Xtrabackup 8.4.0-1, I am hitting a hard crash on Percona Server.

Command used:

xtrabackup --backup --target-dir=/backupdir

The backup fails with this error:

241203 15:45:32 version_check Connected to MySQL server
241203 15:45:32 version_check Executing a version check against the server...
241203 15:45:32 version_check Done.
2024-12-03T15:45:32.449340-00:00 0 [Note] [MY-011825] [Xtrabackup] Connecting to MySQL server host: localhost, user: mysqladm, password: set, port: not set, socket: /home/mysqladm/tmp/mysql.sock
2024-12-03T15:45:32.469364-00:00 0 [Note] [MY-011825] [Xtrabackup] Using server version 8.4.2-2

2024-12-03T15:45:32.482064-00:00 0 [Note] [MY-011825] [Xtrabackup] Executing LOCK TABLES FOR BACKUP ...

xtrabackup: Unknown error 1158

2024-12-03T15:45:32.714657-00:00 0 [ERROR] [MY-011825] [Xtrabackup] failed to execute query 'LOCK TABLES FOR BACKUP' : 2013 (HY000) Lost connection to MySQL server during query

In the mysqld.log I see the follow information:

  what():  basic_string: construction from null is not valid
2024-12-03T15:41:56Z UTC - mysqld got signal 6 ;
Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.
BuildID[sha1]=54e636f0b2e1352bcb36c593f0982176de3d55ef
Server Version: 8.4.2-2 Percona Server (GPL), Release 2, Revision d5292ef1

Thread pointer: 0x7f83f80c1500
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7f84d0725b90 thread_stack 0x100000
 #0 0xe25764 <unknown>
 #1 0xe25d3b <unknown>
 #2 0x7f84e9653d8f <unknown>
 #3 0x7f84e96a054c <unknown>
 #4 0x7f84e9653ce5 <unknown>
 #5 0x7f84e96277f2 <unknown>
 #6 0x7f84e99a1a20 <unknown>
 #7 0x7f84e99ad39b <unknown>
 #8 0x7f84e99ac368 <unknown>
 #9 0x7f84e99acae9 <unknown>
 #10 0x7f84e9819373 <unknown>
 #11 0x7f84e9819a10 <unknown>
 #12 0x7f84e99ad65b <unknown>
 #13 0x7f84e99a42d2 <unknown>
 #14 0x7f84d1d25254 <unknown>
 #15 0x7f84d1d07591 <unknown>
 #16 0x7f84d1d05db9 <unknown>
 #17 0x7f84d1d0a792 <unknown>
 #18 0x7f84d1cc4d3b <unknown>
 #19 0xc61aed <unknown>
 #20 0xe25deb <unknown>
 #21 0xe28475 <unknown>
 #22 0xcd09ca <unknown>
 #23 0xcd4580 <unknown>
 #24 0xcd67d5 <unknown>
 #25 0xcd73f6 <unknown>
 #26 0xe277c6 <unknown>
 #27 0x1a07131 <unknown>
 #28 0x7f84e969e801 <unknown>
 #29 0x7f84e963e44f <unknown>
 #30 0xffffffffffffffff <unknown>

Trying to get some variables.                                                                                                                        
Some pointers may be invalid and cause the dump to abort.                                                                                            
Query (7f83f819e760): LOCK TABLES FOR BACKUP                                                                                                         
Connection ID (thread ID): 11                                                                                                                        
Status: NOT_KILLED

Please let me know if you need further details.

Thanks,

-Wayne

2

Just a bit more information to share.

Looks like when the component_audit_log_filter is installed then Percona Server crashes when we try a backup.

When I remove the component_audit_log_filter like so:

UNINSTALL COMPONENT 'file://component_audit_log_filter';

Backup run with no issues.

Should I be passing any other switch to xtrabackup to account for the audit log filter?

3

Hi Wayne,

If this crash is repeatable with the component_audit_log_filter, please report it as a bug under jira.percona.com.

Should I be passing any other switch to xtrabackup to account for the audit log filter?

Xtrabackup does not have a specific option for the audit log filter. You can explore all available Xtrabackup options on the documentation page linked below.

4

Thank you, we have sorted out the issue.

5

I have been able to recreate this crash in several different systems. All testing has been done using RHEL9 Running the latest version of Percona Server 8.4 and Xtrabackup 8.4.

So here’s whats happening.

Audit log filter component is installed per the documentation.
I setup a filter as prescribed in the documentation.

SELECT audit_log_filter_set_filter('log_all_events', '{ "filter": { "log": true } }');

Then I setup the user

SELECT audit_log_filter_set_user('%', 'log_all_events');

Now we are all setup for auditing. Now I executed this command to do a backup:

sudo /usr/bin/xtrabackup --user=XXXX --password=XXXXX --backup --target-dir=/tmp/backup --socket=/var/lib/mysql/mysql.sock --datadir=/var/lib/mysql

Here is the on screen output:

2025-02-08T16:50:45.190748-05:00 0 [Note] [MY-011825] [Xtrabackup] Connecting to MySQL server host: localhost, user: root, password: set, port: not set, socket: /var/lib/mysql/mysql.sock
2025-02-08T16:50:45.277329-05:00 0 [Note] [MY-011825] [Xtrabackup] Using server version 8.4.3-3
2025-02-08T16:50:45.292581-05:00 0 [Note] [MY-011825] [Xtrabackup] Executing LOCK TABLES FOR BACKUP ...
xtrabackup: Unknown error 1158
2025-02-08T16:50:45.655991-05:00 0 [ERROR] [MY-011825] [Xtrabackup] failed to execute query 'LOCK TABLES FOR BACKUP' : 2013 (HY000) Lost connection to MySQL server during query

Now the information from the mysqld.log:

2025-02-08T21:50:45Z UTC - mysqld got signal 6 ;
Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.
BuildID[sha1]=99b4ce2816ce4597c4c5771312cf4a42f4f410b8
Server Version: 8.4.3-3 Percona Server (GPL), Release 3, Revision 505340da

Thread pointer: 0x7f7418008ad0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7f74604f3b90 thread_stack 0x100000
 #0 0xe29444 <unknown>
 #1 0xe29a1b <unknown>
 #2 0x7f747803e72f <unknown>
 #3 0x7f747808ba6c <unknown>
 #4 0x7f747803e685 <unknown>
 #5 0x7f7478028832 <unknown>
 #6 0x7f74784a1b20 <unknown>
 #7 0x7f74784ad52b <unknown>
 #8 0x7f74784ac4f8 <unknown>
 #9 0x7f74784acc79 <unknown>
 #10 0x7f74786412d3 <unknown>
 #11 0x7f7478641970 <unknown>
 #12 0x7f74784ad7eb <unknown>
 #13 0x7f74784a43d2 <unknown>
 #14 0x7f746921ef04 <unknown>
 #15 0x7f746920a741 <unknown>
 #16 0x7f746920b869 <unknown>
 #17 0x7f746920f942 <unknown>
 #18 0x7f74691caa37 <unknown>
 #19 0xc65aed <unknown>
 #20 0xe29acb <unknown>
 #21 0xe2c155 <unknown>
 #22 0xcd4bda <unknown>
 #23 0xcd8790 <unknown>
 #24 0xcda9e5 <unknown>
 #25 0xcdb606 <unknown>
 #26 0xe2b4a6 <unknown>
 #27 0x1a055a1 <unknown>
 #28 0x7f7478089d21 <unknown>
 #29 0x7f747810ed3f <unknown>
 #30 0xffffffffffffffff <unknown>

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (7f7418176c00): LOCK TABLES FOR BACKUP
Connection ID (thread ID): 9
Status: NOT_KILLED

Now if I drop all the data from the mysql.audit_log_user table, backups will run without issue.

6

I have also found that this issues also impacts the full SST of PXC 8.4.3 when SST method is xtrabackup-v2.

7

I can prevent the crash by adding the below stanza to the my.cnf.

[xtrabackup]
lock-ddl=off
lock-ddl-per-table=1

[sst]
inno-backup-opts=“–lock-ddl-per-table --lock-ddl=OFF”

These option will allow me to work around the crash of Percona Server 8.4 when the backup runs or full SST take place, but how much risk am I really adding to the stability of my backups and the SST process.

8

Hello Wayne,
I believe you were hitting this bug: PS-9668
It was fixed in Percona Server version 8.4.4, and the same fix will be included in PXC 8.4.4, but this one will be released bit later.