Query regarding Xtrabackup issue

Hi,
I am doing POC on Kubernetes. I am Following below links for setup master/slave setup using MySQL & xtrabackup.
Running below issue: In our env. we have multiple DC and in. order to communicate one DC to another we have to use kube load balancer service. I plan is to setup one master on one DC and slave on another DC and they will communicate via load balancer.  I have tested the setup but getting below error.

===

➜  working k logs -f -c xtrabackup mysql-master-5b67c98658-z4vck

+ cd /var/lib/mysql

+ exec ncat --listen --keep-open --send-only --max-conns=1 3307 -c 'xtrabackup --backup --slave-info --stream=xbstream --host=127.0.0.1 --user=root --password=root'

200407 19:16:32  version_check Connecting to MySQL server with DSN 'dbi:mysql:;mysql_read_default_group=xtrabackup;host=127.0.0.1;port=3306;mysql_socket=/var/run/mysqld/mysqld.sock' as 'root'  (using password: YES).

200407 19:16:32  version_check Connected to MySQL server

200407 19:16:32  version_check Executing a version check against the server...

200407 19:16:32  version_check Done.

200407 19:16:32 Connecting to MySQL server host: 127.0.0.1, user: root, password: set, port: 3306, socket: /var/run/mysqld/mysqld.sock

Using server version 5.7.29-log

xtrabackup version 2.4.4 based on MySQL server 5.7.13 Linux (x86_64) (revision id: df58cf2)

xtrabackup: uses posix_fadvise().

xtrabackup: cd to /var/lib/mysql

xtrabackup: open files limit requested 0, set to 1048576

xtrabackup: using the following InnoDB configuration:

xtrabackup:   innodb_data_home_dir = .

xtrabackup:   innodb_data_file_path = ibdata1:12M:autoextend

xtrabackup:   innodb_log_group_home_dir = ./

xtrabackup:   innodb_log_files_in_group = 2

xtrabackup:   innodb_log_file_size = 50331648

InnoDB: Number of pools: 1

200407 19:16:32 >> log scanned up to (12480867)

xtrabackup: Generating a list of tablespaces

InnoDB: Allocated tablespace ID 2 for mysql/plugin, old maximum was 0

200407 19:16:32 [01] Streaming ./ibdata1

xtrabackup: Error writing file 'UNOPENED' (Errcode: 32 - Broken pipe)

xb_stream_write_data() failed.

xtrabackup: Error writing file 'UNOPENED' (Errcode: 32 - Broken pipe)

[01] xtrabackup: Error: xtrabackup_copy_datafile() failed.

[01] xtrabackup: Error: failed to copy datafile.

Ncat: assertion failed: count <= INT_MAX QUITTING.


It's working fine with normal  cluster IP service.



Hi,
This does not look like Percona Xtrabackup issue but rather network issue in your environment.     I’d check if using same ncat the same way with large static file creates same problems

Ok, Thanks for the update. Now I am running into this below issue:
1. I am starting stream backup so that I can easily  scale up my slaves in pod on kubernetes env. 

 # Start a server to send backups when requested by peers.

          exec ncat --listen --keep-open --send-only --max-conns=1 3307 -c \

            "xtrabackup --backup --slave-info --stream=xbstream --host=127.0.0.1 --user=root"


but I am getting below warnings on mysql error logs continuously. Is there any workaround of it.

2020-04-11T08:50:26.233510Z 8 [Note] Got an error reading communication packets

2020-04-11T08:50:26.308221Z 9 [Note] Got an error reading communication packets

2020-04-11T08:50:31.233307Z 12 [Note] Got an error reading communication packets

2020-04-11T08:50:31.308479Z 13 [Note] Got an error reading communication packets

2020-04-11T08:50:36.233945Z 19 [Note] Got an error reading communication packets

2020-04-11T08:50:36.308112Z 20 [Note] Got an error reading communication packets

Following below link for setup master/slaves.


https://kubernetes.io/docs/tasks/run-application/run-replicated-stateful-application/