State Transfer Problem

[finnthehuman]

Percona XtraDB Cluster from the Percona repo (percona-xtradb-cluster-server-5.5/5.5.27-23.6-356.oneiric) on Ubuntu 11.10 64-bit.

Error is “[ERROR] WSREP: async IST sender failed to serve tcp://x.x.x.x:4568: ist send failed: 1’, asio error ‘Connection reset by peer’: 104 (Connection reset by peer) at galera/src/ist.cpp:send():740”. Node fails to fall back to full state transfer and mysql service fails to start. No innobackup or sst error logs are created after failure.

[revin]

@finnthehuman,

When the original problem happens again, can you post the contents of the innobackup.backup.log from the donor. This is found within the data directory.

[revin]

@finnthehuman,

From this log on the donor http://pastebin.com/YVinfUeM I see the below, so it may have the log file available. Let us know what you find out

130513 19:01:41 [ERROR] WSREP: Failed to read from: wsrep_sst_xtrabackup 'donor' '10.7.3.53:4444/xtrabackup_sst' '(null)' '/var/lib/mysql/' '/etc/mysql/my.cnf' '3214b7d4-8b59-11e2-0800-fde78b56d029' '109567765' '1'
130513 19:01:41 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup 'donor' '10.7.3.53:4444/xtrabackup_sst' '(null)' '/var/lib/mysql/' '/etc/mysql/my.cnf' '3214b7d4-8b59-11e2-0800-fde78b56d029' '109567765' '1': 1 (Operation not permitted)
130513 19:01:41 [Warning] WSREP: 0 (db-2): State transfer to 1 (db-3) failed: -1 (Operation not permitted)

[finnthehuman]

After upgrade to 5.5.30-rel30.2 initial start/joining the cluster state transfer failed with “InnoDB: Error: io_setup() failed with EAGAIN after 5 attempts.” More information is available at this URL: http://elenst.ru/pensieve/it-bits/ai…in-particular/. Increased the fs.aio-max-nr in sysctl.conf and the transfer worked on the next try.

[finnthehuman]

Just wanted to say that I have upgraded all of my nodes/clusters to 5.5.30-rel30.2 I have not had a single problem since so there is clearly something wrong with the old version. If anyone is still running on 5.5.27 you should probably upgrade as soon as you can.