Xtrabackup fails mysqld got signal 6

Hello everyone,

I’m trying to run xtrabackup. the command

xtrabackup --backup \
    --no-version-check \
    --compress=lz4 \
    --compress-threads=2 \
    --parallel=2 \
    --read-buffer-size=500Mb \
    --datadir="/var/lib/mysql/" \
    --target_dir="/data/tmp" \
    --stream=xbstream

The backup fails with ‘got signal 6’ error
traceback:

2024-11-10T09:28:00.307584+10:00 0 [Note] [MY-011825] [Xtrabackup] recognized server arguments: --datadir=/var/lib/mysql --open_files_limit=65535 --innodb_buffer_pool_size=184G --innodb_flus
h_method=O_DIRECT --innodb_log_files_in_group=16 --innodb_log_file_size=2048M --innodb_log_group_home_dir=/var/lib/mysql_iblog/ --innodb_flush_log_at_trx_commit=2 --innodb_file_per_table=1 -
-innodb_log_buffer_size=64M --innodb_read_io_threads=8 --innodb_write_io_threads=8 --log_bin=/var/lib/mysql_logbin/log_bin --server-id=60 --datadir=/var/lib/mysql/ 
2024-11-10T09:28:00.307892+10:00 0 [Note] [MY-011825] [Xtrabackup] recognized client arguments: --password=* --backup=1 --no-version-check=1 --compress=lz4 --compress-threads=2 --parallel=2 
--read_buffer_size=500Mb --target-dir=/data/tmp --stream=xbstream                        
xtrabackup version 8.0.35-31 based on MySQL server 8.0.35 Linux (x86_64) (revision id: 55ec21d7)
2024-11-10T09:28:00.307942+10:00 0 [Note] [MY-011825] [Xtrabackup] Connecting to MySQL server host: localhost, user: not set, password: set, port: not set, socket: not set
2024-11-10T09:28:00.315494+10:00 0 [Note] [MY-011825] [Xtrabackup] Using server version 8.0.33-25
2024-11-10T09:28:00.317846+10:00 0 [Note] [MY-011825] [Xtrabackup] Executing LOCK TABLES FOR BACKUP ...
2024-11-10T09:28:00.322149+10:00 0 [Note] [MY-011825] [Xtrabackup] uses posix_fadvise().
2024-11-10T09:28:00.322200+10:00 0 [Note] [MY-011825] [Xtrabackup] cd to /var/lib/mysql/
2024-11-10T09:28:00.322236+10:00 0 [Note] [MY-011825] [Xtrabackup] open files limit requested 65535, set to 65535
2024-11-10T09:28:00.326195+10:00 0 [Note] [MY-011825] [Xtrabackup] using the following InnoDB configuration:
2024-11-10T09:28:00.326238+10:00 0 [Note] [MY-011825] [Xtrabackup] innodb_data_home_dir = .                                                                                                   
2024-11-10T09:28:00.326246+10:00 0 [Note] [MY-011825] [Xtrabackup] innodb_data_file_path = ibdata1:12M:autoextend
2024-11-10T09:28:00.326314+10:00 0 [Note] [MY-011825] [Xtrabackup] innodb_log_group_home_dir = /var/lib/mysql_iblog/
2024-11-10T09:28:00.326340+10:00 0 [Note] [MY-011825] [Xtrabackup] innodb_log_files_in_group = 16
2024-11-10T09:28:00.326349+10:00 0 [Note] [MY-011825] [Xtrabackup] innodb_log_file_size = 2147483648
2024-11-10T09:28:00.326388+10:00 0 [Note] [MY-011825] [Xtrabackup] using O_DIRECT
2024-11-10T09:28:00.328339+10:00 0 [Note] [MY-011825] [Xtrabackup] inititialize_service_handles suceeded
2024-11-10T09:28:00.455265+10:00 0 [Note] [MY-011825] [Xtrabackup] Connecting to MySQL server host: localhost, user: not set, password: set, port: not set, socket: not set
2024-11-10T09:28:00.461341+10:00 0 [Note] [MY-011825] [Xtrabackup] Redo Log Archiving is not set up.
2024-11-10T09:28:00.576047+10:00 0 [Note] [MY-011825] [Xtrabackup] Starting to parse redo log at lsn = 64635836936287                                              
2024-11-10T09:28:00.577756+10:00 0 [Note] [MY-012564] [InnoDB] Recovery parsing buffer extended to 4194304.                                                                                   2024-11-10T09:28:00.579519+10:00 0 [Note] [MY-012564] [InnoDB] Recovery parsing buffer extended to 8388608.
2024-11-10T09:28:00.582823+10:00 0 [Note] [MY-012564] [InnoDB] Recovery parsing buffer extended to 16777216.                                                                                  2024-11-10T09:28:00.589304+10:00 0 [Note] [MY-012564] [InnoDB] Recovery parsing buffer extended to 33554432.                                    
2024-11-09T23:28:00Z UTC - mysqld got signal 6 ;                                                              
Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.                                                                                                
BuildID[sha1]=                                                                                                                                                                                Thread pointer: 0x0                                                                            
Attempting backtrace. You can use the following information to find out                                                                                                                       
where mysqld died. If you see no messages after this, something went                                                                                                                          
terribly wrong...                                                                                                                                                                             
stack_bottom = 0 thread_stack 0x100000                                                                                                                                                        
xtrabackup(my_print_stacktrace(unsigned char const*, unsigned long)+0x41) [0x18df6f1]                                                                                                         
xtrabackup(print_fatal_signal(int)+0x3bc) [0xddf5bc]
xtrabackup(handle_fatal_signal+0x95) [0xddf665] 
/lib64/libc.so.6(+0x3e6f0) [0x7f33b383e6f0]
/lib64/libc.so.6(+0x8b94c) [0x7f33b388b94c]
/lib64/libc.so.6(raise+0x16) [0x7f33b383e646]
/lib64/libc.so.6(abort+0xd3) [0x7f33b38287f3]
xtrabackup() [0x8a1786]
xtrabackup() [0x8a9ba7]
xtrabackup(Redo_Log_Writer::write_buffer(unsigned char*, unsigned long)+0x1a4) [0x905c74]
xtrabackup(Redo_Log_Data_Manager::start()+0xcd) [0x916dfd]
xtrabackup(xtrabackup_backup_func()+0x958) [0x8cf408]
xtrabackup(main+0x14ac) [0x871eec]
/lib64/libc.so.6(+0x29590) [0x7f33b3829590]
/lib64/libc.so.6(__libc_start_main+0x80) [0x7f33b3829640]
xtrabackup(_start+0x25) [0x89e195]

Please report a bug at https://jira.percona.com/projects/PXB

Could someone help to debug and solve the issue?

Hey @PavelS,
Could you please enable core dumps on your OS and provide one when PXB crashes again? That would really help out the developers in debugging the issue.

Hey @matthewb, thanks for the reply. How can I enable coredumps?

Depends on your OS. Quick google search should reveal many blogs/HOWTOs for enabling on different OSes.

I have found a workaround for the issue. Actually, even two.

  1. I’ve tried to run perfectly the same backup command using docker xtrabackup image. The version of the image was the same as for xtrabackup installed in the system. It works perfectly and this is weird. At a first glance there are no difference between two setups except of the docker.

  2. I’ve managed to run the backup after changing compress method lz4->zstd. It works fine as well

I don’t have any conclusions at the moment