Xtrabackup: Error writing file 'UNOPENED' (Errcode: 32 - Broken pipe) When streaming to S3

MySQL & MariaDB Percona XtraBackup
1

First off, here’s my command

xtrabackup --backup --stream=xbstream --target-dir=/backup --extra-lsndir=/backup | xbcloud put s3:///initial

I’m trying to stream to s3. If I run the initial xtrabackup command and remove the “–stream” argument, I can write the full backup to a local volume without issue. When I add the stream argument and pipe it to xbcloud, I consistently get the same error:

[root@ip-172-31-120-241 backup]# xtrabackup --backup --stream=xbstream --target-dir=/backup --extra-lsndir=/backup | xbcloud put s3:///initial

xtrabackup: recognized server arguments: --datadir=/var/lib/mysql --tmpdir=/var/lib/mysqltmp --innodb_flush_method=O_DIRECT --innodb_flush_log_at_trx_commit=2 --innodb=FORCE --innodb_buffer_pool_size=90G --innodb_file_per_table=1 --innodb_read_io_threads=64 --innodb_write_io_threads=64 --server-id=8675309 --log_bin=/var/lib/mysqllogs/bin-log

xtrabackup: recognized client arguments: --user=root --password=* --backup=1 --stream=xbstream --target-dir=/backup --extra-lsndir=/backup

201220 14:17:43 S3 access key is not specified

201220 14:17:43 version_check Connecting to MySQL server with DSN ‘dbi:mysql:;mysql_read_default_group=xtrabackup’ as ‘root’ (using password: YES).

201220 14:17:43 version_check Connected to MySQL server

201220 14:17:43 version_check Executing a version check against the server…

201220 14:17:43 version_check Done.

201220 14:17:43 Connecting to MySQL server host: localhost, user: root, password: set, port: not set, socket: not set

Using server version 5.6.50-log

xtrabackup version 2.4.21 based on MySQL server 5.7.32 Linux (x86_64) (revision id: 5988af5)

xtrabackup: uses posix_fadvise().

xtrabackup: cd to /var/lib/mysql

xtrabackup: open files limit requested 0, set to 65535

xtrabackup: using the following InnoDB configuration:

xtrabackup: innodb_data_home_dir = .

xtrabackup: innodb_data_file_path = ibdata1:12M:autoextend

xtrabackup: innodb_log_group_home_dir = ./

xtrabackup: innodb_log_files_in_group = 2

xtrabackup: innodb_log_file_size = 50331648

xtrabackup: using O_DIRECT

InnoDB: Number of pools: 1

201220 14:17:43 >> log scanned up to (96714319531924)

xtrabackup: Generating a list of tablespaces

InnoDB: Allocated tablespace ID 6 for Sub_Prime_Consumers/bankinfo, old maximum was 0

201220 14:17:43 [01] Streaming ./ibdata1

xtrabackup: Error writing file ‘UNOPENED’ (Errcode: 32 - Broken pipe)

xb_stream_write_data() failed.

xtrabackup: Error writing file ‘UNOPENED’ (Errcode: 32 - Broken pipe)

[01] xtrabackup: Error: xtrabackup_copy_datafile() failed.

[01] xtrabackup: Error: failed to copy datafile.

201220 14:17:44 >> log scanned up to (96714319550515)

I see an error for s3 access key not specified. I’m relying on S3 roles currently, do I need to set up a key for this to work?

1 Like
2

To answer my own question in case anyone sees this in the future:

I had to create an IAM user, instead of assuming a role on the instance this was run on. Then, I had to specify a region as I was not using us-east-1 (the default).

It seems to be working normally now.

1 Like
3

Hello @insight_warren,

I am experiencing something similar. In my xtrabackup command however, I am entering the --s3-access-key and --s3-secret-access-key values for the IAM user, but still running into the same error. Please advise if you can:

xtrabackup --backup --stream=xbstream --extra-lsndir=/backup --target-dir=/backup | xbcloud put --storage=s3 --s3-endpoint=‘value’ --s3-access-key=‘value’ --s3-secret-access-key=‘value’ --s3-bucket= ‘value’

1 Like
4

You may need to specify region as well. I ended up putting everything I could in my.cnf but I believe region is required as well.

3 Likes
5

@insight_warren is right. Usually when this happens with S3 the solution is to specify region.

1 Like
7

did you resolve this problem？

I have specify region，still error：
xtrabackup: Error writing file ‘UNOPENED’ (Errcode: 32 - Broken pipe)。

8

Hi @blue.li ,

Those errors are normally with credentials. Try to run xbcoud alone like:

xbcloud put...

Do it until you get a message saying it can connect successfully with object storage.