Hi,
I am seeing an error with xtrabackup using xbstream. Here is the error:
2022-10-07T10:36:04.081750-00:00 2 [Note] [MY-011825] [Xtrabackup] Streaming /home/mysqld/DB/data/DB_1.dbf
xtrabackup: Error writing file ‘’ (OS errno 32 - Broken pipe)
xb_stream_write_data() failed.
2022-10-07T10:36:04.871895-00:00 1 [Note] [MY-011825] [Xtrabackup] >> log scanned up to (156528488)
xtrabackup: Error writing file ‘’ (OS errno 32 - Broken pipe)
2022-10-07T10:36:04.894870-00:00 2 [ERROR] [MY-011825] [Xtrabackup] xtrabackup_copy_datafile() failed
2022-10-07T10:36:04.894895-00:00 2 [ERROR] [MY-011825] [Xtrabackup] failed to copy datafile /home/mysqld/DB/data/DB_1.dbf
PID trace:
— SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0x60} —
+++ killed by SIGSEGV +++
DB version: Percona Server 8.0.29
1 Like
Hi @Chanakya,
Thank you for being part of the Percona Community. Could you post here the command you are executing? Also, the version on Percona XtraBackup.
This is a Segmentation Fault error: Segmentation fault - Wikipedia
We need to find the causes of this error, but looks like xbstream is crashing and xtrabackup can’t
keep on being streamed.
Thanks,
Pep
1 Like
Hi @Chanakya ,
One thing that caught my attention is the file extension pxb is streaming (DBF) this doesn’t seems to be a file that MySQL would create.
Does the crash always happens on the same file?
As @Pep_Pla mentioned, can you please send us a full error log and also clarify on this .dbf file. Do you know which file is it and does the crash happens always on this particular file?
Thanks in advance.
2 Likes
Hi @Marcelo_Altmann @Pep_Pla Thank you for your time reviewing the error: “.dbf” is the innodb_data_file_path . Also I am pasting log and also my.cnf, Please help.
2022-10-07T10:36:03.425576-00:00 0 [Note] [MY-011825] [Xtrabackup] recognized server arguments: --datadir=/home/mysqld/DBNAME/data --log_bin=DBNAME_b_binlog --server-id=2 --innodb_data_home_dir=/home/mysqld/DBNAME/data --innodb_data_file_path=DBNAME_1.dbf:1024M;DBNAME_2.dbf:500M:autoextend --innodb_buffer_pool_size=500M --innodb_flush_log_at_trx_commit=1 --innodb_log_group_home_dir=/home/mysqld/DBNAME/data --innodb_log_buffer_size=64M --innodb_log_file_size=512M --innodb_log_files_in_group=2 --innodb_flush_method=O_DIRECT --innodb_open_files=2048 --innodb_use_native_aio=0 --tmpdir=/home/mysqld/DBNAME/tmp
2022-10-07T10:36:03.425816-00:00 0 [Note] [MY-011825] [Xtrabackup] recognized client arguments: --port=3464 --socket=/tmp/mysqld_DBNAME.sock --backup=1 --user=root --password=* --stream=xbstream --extra-lsndir=/mydata/DBNAME/backup/MYBACKUP--DBNAME-ms11p01nt-mysqldb002_1665138962/config --slave-info=1 --target-dir=/mydata/DBNAME/backup/MYBACKUP--DBNAME-ms11p01nt-mysqldb002_1665138962/database --read_buffer_size=400M --no-server-version-check=1 --strict=0
xtrabackup version 8.0.29-22 based on MySQL server 8.0.29 Linux (x86_64) (revision id: c31e7ddcce3)
221007 10:36:03 version_check Connecting to MySQL server with DSN 'dbi:mysql:;mysql_read_default_group=xtrabackup;port=3464;mysql_socket=/tmp/mysqld_DBNAME.sock' as 'root' (using password: YES).
Failed to connect to MySQL server as DBD::mysql module is not installed at - line 1548.
2022-10-07T10:36:03.496889-00:00 0 [Note] [MY-011825] [Xtrabackup] Connecting to MySQL server host: localhost, user: root, password: set, port: 3464, socket: /tmp/mysqld_DBNAME.sock
2022-10-07T10:36:03.503713-00:00 0 [Note] [MY-011825] [Xtrabackup] Using server version 8.0.29-21
2022-10-07T10:36:03.508908-00:00 0 [Note] [MY-011825] [Xtrabackup] Executing LOCK TABLES FOR BACKUP ...
2022-10-07T10:36:03.510659-00:00 0 [Note] [MY-011825] [Xtrabackup] uses posix_fadvise().
2022-10-07T10:36:03.510702-00:00 0 [Note] [MY-011825] [Xtrabackup] cd to /home/mysqld/DBNAME/data
2022-10-07T10:36:03.510717-00:00 0 [Note] [MY-011825] [Xtrabackup] open files limit requested 0, set to 131072
2022-10-07T10:36:03.510792-00:00 0 [Note] [MY-011825] [Xtrabackup] using the following InnoDB configuration:
2022-10-07T10:36:03.510801-00:00 0 [Note] [MY-011825] [Xtrabackup] innodb_data_home_dir = /home/mysqld/DBNAME/data
2022-10-07T10:36:03.510807-00:00 0 [Note] [MY-011825] [Xtrabackup] innodb_data_file_path = DBNAME_1.dbf:1024M;DBNAME_2.dbf:500M:autoextend
2022-10-07T10:36:03.510858-00:00 0 [Note] [MY-011825] [Xtrabackup] innodb_log_group_home_dir = /home/mysqld/DBNAME/data
2022-10-07T10:36:03.510865-00:00 0 [Note] [MY-011825] [Xtrabackup] innodb_log_files_in_group = 2
2022-10-07T10:36:03.510876-00:00 0 [Note] [MY-011825] [Xtrabackup] innodb_log_file_size = 536870912
2022-10-07T10:36:03.510900-00:00 0 [Note] [MY-011825] [Xtrabackup] using O_DIRECT
2022-10-07T10:36:03.511622-00:00 0 [Note] [MY-013251] [InnoDB] Number of pools: 1
2022-10-07T10:36:03.512400-00:00 0 [Note] [MY-011825] [Xtrabackup] inititialize_service_handles suceeded
2022-10-07T10:36:03.770713-00:00 0 [Note] [MY-011825] [Xtrabackup] Connecting to MySQL server host: localhost, user: root, password: set, port: 3464, socket: /tmp/mysqld_DBNAME.sock
2022-10-07T10:36:03.776632-00:00 0 [Note] [MY-011825] [Xtrabackup] Redo Log Archiving is not set up.
2022-10-07T10:36:03.871144-00:00 0 [Note] [MY-012529] [InnoDB] Redo log format is v4. The redo log was created before MySQL 8.0.3.
2022-10-07T10:36:03.871194-00:00 0 [Note] [MY-012529] [InnoDB] Redo log format is v4. The redo log was created before MySQL 8.0.3.
2022-10-07T10:36:03.871396-00:00 0 [Note] [MY-011825] [Xtrabackup] Starting to parse redo log at lsn = 156524102
2022-10-07T10:36:03.871716-00:00 1 [Note] [MY-011825] [Xtrabackup] >> log scanned up to (156528488)
2022-10-07T10:36:04.075329-00:00 0 [Note] [MY-011825] [Xtrabackup] Generating a list of tablespaces
2022-10-07T10:36:04.075422-00:00 0 [Note] [MY-011825] [Xtrabackup] Generating a list of tablespaces
2022-10-07T10:36:04.075437-00:00 0 [Note] [MY-012204] [InnoDB] Scanning './'
2022-10-07T10:36:04.078634-00:00 0 [Note] [MY-012208] [InnoDB] Completed space ID check of 2 files.
2022-10-07T10:36:04.078823-00:00 0 [Warning] [MY-012091] [InnoDB] Allocated tablespace ID 3 for mysql/gsd_monitor, old maximum was 0
2022-10-07T10:36:04.079049-00:00 0 [Note] [MY-013252] [InnoDB] Using undo tablespace './undo_001'.
2022-10-07T10:36:04.079527-00:00 0 [Note] [MY-013252] [InnoDB] Using undo tablespace './undo_002'.
2022-10-07T10:36:04.080433-00:00 0 [Note] [MY-012910] [InnoDB] Opened 2 existing undo tablespaces.
2022-10-07T10:36:04.081750-00:00 2 [Note] [MY-011825] [Xtrabackup] Streaming /home/mysqld/DBNAME/data/DBNAME_1.dbf
xtrabackup: Error writing file '<unopen fd>' (OS errno 32 - Broken pipe)
xb_stream_write_data() failed.
2022-10-07T10:36:04.871895-00:00 1 [Note] [MY-011825] [Xtrabackup] >> log scanned up to (156528488)
xtrabackup: Error writing file '<unopen fd>' (OS errno 32 - Broken pipe)
2022-10-07T10:36:04.894870-00:00 2 [ERROR] [MY-011825] [Xtrabackup] xtrabackup_copy_datafile() failed
2022-10-07T10:36:04.894895-00:00 2 [ERROR] [MY-011825] [Xtrabackup] failed to copy datafile /home/mysqld/DBNAME/data/DBNAME_1.dbf
[client]
port=3464
socket=/tmp/mysqld_DBNAME.sock
[mysqld]
port=3464
bind-address=ms11p01nt-mysqldb002.ise.apple.com
socket=/tmp/mysqld_DBNAME.sock
pid-file=DBNAME_b.pid
datadir=/home/mysqld/DBNAME/data
basedir=/home/mysqld/sys/8.0.29_percona
mysqlx=OFF
binlog_format=mixed
log-bin=DBNAME_b_binlog
server-id=2
gtid_mode=ON
# enforce-gtid-consistency=true
enforce-gtid-consistency=ON
sync_binlog=1
#expire_logs_days=7
transaction-isolation=READ-COMMITTED
#character-sets-dir=/home/mysqld/sys/8.0.29_percona/share/mysql/charsets
character-set-server=utf8mb4
collation-server=utf8mb4_unicode_ci
#skip-character-set-client-handshake
event_scheduler=ON
relay-log=/home/mysqld/DBNAME/data/DBNAME_b_relaylog
log-slave-updates
log_error=/home/mysqld/DBNAME/data/DBNAME_b.err
log_error_verbosity=2
back_log=128
long_query_time=15
slow_query_log=1
slow_query_log_file=DBNAME_b_slow.log
master-info-repository=TABLE
replicate-ignore-table=mysql.gsd_runs
replicate-ignore-table=mysql.gsd_monitor
replicate-ignore-table=mysql.gsd_tsmstate
max_relay_log_size=100M
query_prealloc_size = 65536
max_connections = 400
max_connect_errors = 50000
innodb_data_home_dir=/home/mysqld/DBNAME/data
innodb_data_file_path=DBNAME_1.dbf:1024M;DBNAME_2.dbf:500M:autoextend
innodb_buffer_pool_size=500M
innodb_flush_log_at_trx_commit=1
innodb_fast_shutdown=0
innodb_log_group_home_dir=/home/mysqld/DBNAME/data
innodb_log_buffer_size = 64M
innodb_log_file_size = 512M
innodb_log_files_in_group=2
innodb_flush_method=O_DIRECT
innodb_thread_concurrency=16
innodb_open_files=2048
innodb_lock_wait_timeout=60
key_buffer_size=128M
max_heap_table_size=128M
tmp_table_size=128M
max_allowed_packet=128M
sort_buffer_size=8M
myisam_sort_buffer_size=16M
read_rnd_buffer_size=8M
#sql_mode=""
innodb_use_native_aio=0
tmpdir=/home/mysqld/DBNAME/tmp
### To enable the server for encrypted connections
# ssl-ca=/home/mysqld/etc/ssl/ca-cert.pem
# ssl-cert=/home/mysqld/etc/ssl/server-cert.pem
# ssl-key=/home/mysqld/etc/ssl/server-key.pem
### Clients are required to use encrypted connections
# require_secure_transport=ON
read-only
##skip-slave-start=1
#Audit Settings
audit_log=ON
audit_log_file='/home/mysqld/DBNAME/log/audit/DBNAME_b_audit.log'
audit_log_format=JSON
audit_log_rotate_on_size=500M
audit_log_strategy=ASYNCHRONOUS
audit_log_policy=ALL
audit_log_rotations=4
audit_log_exclude_commands='commit,insert,insert_select,purge,purge_before_date,replace_select,replace,select,update,update_multi,truncate,xa_end,xa_commit,xa_prepare,xa_recover,xa_rollback,xa_start,rollback_to_savepoint,rollback'
#Password Settings
plugin-load-add=validate_password.so
validate-password=FORCE_PLUS_PERMANENT
1 Like
Hi @Chanakya,
Could you paste here the exact commands you are using? You are streaming the backup, so it would be great to know also the exact sequence of commands executed.
Thank you!
Pep
1 Like
Hi @chanakya.
Thanks for clarifying that the .dbf is your ibdata file.
As mentioned before, the error is always coming on the same file, however, this is the first file xtrabackup is trying to stream, so the issue might lie on the other end - xbstream.
If you can reproduce the issue, can you please run xbstream under strace and send us the result file tmp/xbstream.strace:
xtrabackup --backup --stream=xbstream ... | strace -f -s8192 -ttt -o/tmp/xbstream.strace xbstream ...
Thanks in advance.
1 Like
Hi @Marcelo_Altmann @Pep_Pla
xtrabackup --defaults-file=/home/mysqld/DB/etc/my.cnf --verbose --backup --user=root --password=$PWD --stream=xbstream --extra-lsndir=/mydata/DB/backup/MYBACKUP–DB_1664994330/config --slave-info --target-dir=/mydata/DB/backup/MYBACKUP–DB_1664994330/database --skip-strict | xbcloud put --verbose --storage=s3 --s3-endpoint=‘https://s3:xxxx’ --s3-access-key=‘’ --s3-secret-key=‘’ --s3-bucket-lookup=PATH --s3-bucket=‘mysql-DB-production-month’
This is a sample. And I tried under strace, file is coming empty.
1 Like
Hi @Marcelo_Altmann @Pep_Pla Can you please help with above. This has been a blocker in our env.
1 Like
HI strace should not come empty.
first check if it is installed:
which strace
Then can you please run it again?
strace -f -s8192 -ttt -o/tmp/xtrabackup.strace xtrabackup --defaults-file=/home/mysqld/DB/etc/my.cnf --verbose --backup --user=root --password=$PWD --stream=xbstream --extra-lsndir=/mydata/DB/backup/MYBACKUP–DB_1664994330/config --slave-info --target-dir=/mydata/DB/backup/MYBACKUP–DB_1664994330/database --skip-strict | strace -f -s8192 -ttt -o/tmp/xbcloud.strace xbcloud put --verbose --storage=s3 --s3-endpoint=‘https://s3:xxxx’ --s3-access-key=‘’ --s3-secret-key=‘’ --s3-bucket-lookup=PATH --s3-bucket=‘mysql-DB-production-month’
Please send us the full output of those commands and also /tmp/xbcloud.strace & /tmp/xtrabackup.strace
1 Like