Not the answer you need?
Register and ask your own question!

Can start mysql or resync SST after last update

PercUsePercUse ContributorInactive User Role Beginner
Hi Guys,

We just updated our mysql cluster from 5.6.26-25.12-1.precise to 5.6.27-25.13-1.precise

Package involved:

percona-xtradb-cluster-client-5.6 (5.6.26-25.12-1.precise => 5.6.27-25.13-1.precise)
percona-xtradb-cluster-common-5.6 (5.6.26-25.12-1.precise => 5.6.27-25.13-1.precise)
percona-xtradb-cluster-galera-3.x (3.12.2-1.precise => 3.13-1.precise)
percona-xtradb-cluster-server-5.6 (5.6.26-25.12-1.precise => 5.6.27-25.13-1.precise)

Since the update, we have been able to start MySQL on 2 nodes but the third one don't want to start or sync
/etc/init.d/mysql start
* Starting MySQL (Percona XtraDB Cluster) database server mysqld

160111 15:15:03 mysqld_safe Starting mysqld daemon with databases from /srv/mysql/data
160111 15:15:03 mysqld_safe WSREP: Running position recovery with --log_error='/srv/mysql/data/wsrep_recovery.I1fypn' --pid-file='/srv/mysql/data/server2-recover.pid'
2016-01-11 15:15:05 0 [Note] /usr/sbin/mysqld (mysqld 5.6.27-76.0-56) starting as process 5251 ...
160111 15:15:48 mysqld_safe WSREP: Recovered position 51c9a3db-aeaa-11e3-a5e6-bffb1faf3524:1305553
Log of wsrep recovery (--wsrep-recover):
2016-01-11 15:15:05 5251 [Note] Plugin 'FEDERATED' is disabled.
InnoDB: Warning: innodb_log_block_size has been changed from default value 512. (###EXPERIMENTAL### operation)
InnoDB: The log block size is set to 4096.
2016-01-11 15:15:05 5251 [Note] InnoDB: Using atomics to ref count buffer pool pages
2016-01-11 15:15:05 5251 [Note] InnoDB: The InnoDB memory heap is disabled
2016-01-11 15:15:05 5251 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2016-01-11 15:15:05 5251 [Note] InnoDB: Memory barrier is not used
2016-01-11 15:15:05 5251 [Note] InnoDB: Compressed tables use zlib 1.2.3.4
2016-01-11 15:15:05 5251 [Note] InnoDB: Using Linux native AIO
2016-01-11 15:15:05 5251 [Note] InnoDB: Using CPU crc32 instructions
2016-01-11 15:15:05 5251 [Note] InnoDB: Initializing buffer pool, size = 10.0G
2016-01-11 15:15:06 5251 [Note] InnoDB: Completed initialization of buffer pool
2016-01-11 15:15:06 5251 [Note] InnoDB: Highest supported file format is Barracuda.
2016-01-11 15:15:45 5251 [Note] InnoDB: 128 rollback segment(s) are active.
2016-01-11 15:15:45 5251 [Note] InnoDB: Waiting for purge to start
2016-01-11 15:15:45 5251 [Warning] InnoDB: Setting thread 5395 nice to -10 failed, current nice 0, errno 13
2016-01-11 15:15:45 5251 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.27-76.0 started; log sequence number 5680797833
2016-01-11 15:15:45 5251 [Warning] InnoDB: Skipping buffer pool dump/restore during wsrep recovery.
2016-01-11 15:15:45 5251 [Warning] InnoDB: Setting thread 5396 nice to -10 failed, current nice 0, errno 13
2016-01-11 15:15:45 5251 [Note] RSA private key file not found: /srv/mysql/data//private_key.pem. Some authentication plugins will not work.
2016-01-11 15:15:45 5251 [Note] RSA public key file not found: /srv/mysql/data//public_key.pem. Some authentication plugins will not work.
2016-01-11 15:15:45 5251 [Note] Server hostname (bind-address): '192.168.10.12'; port: 3306
2016-01-11 15:15:45 5251 [Note] - '192.168.10.12' resolves to '192.168.10.12';
2016-01-11 15:15:45 5251 [Note] Server socket created on IP: '192.168.10.12'.
2016-01-11 15:15:45 5251 [Note] WSREP: Recovered position: 51c9a3db-aeaa-11e3-a5e6-bffb1faf3524:1305553
2016-01-11 15:15:45 5251 [Note] Binlog end
2016-01-11 15:15:45 5251 [Note] Shutting down plugin 'partition'
2016-01-11 15:15:45 5251 [Note] Shutting down plugin 'BLACKHOLE'
2016-01-11 15:15:45 5251 [Note] Shutting down plugin 'ARCHIVE'
2016-01-11 15:15:45 5251 [Note] Shutting down plugin 'PERFORMANCE_SCHEMA'
2016-01-11 15:15:45 5251 [Note] Shutting down plugin 'INNODB_CHANGED_PAGES'
2016-01-11 15:15:45 5251 [Note] Shutting down plugin 'INNODB_SYS_DATAFILES'
2016-01-11 15:15:45 5251 [Note] Shutting down plugin 'INNODB_SYS_TABLESPACES'
2016-01-11 15:15:45 5251 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN_COLS'
2016-01-11 15:15:45 5251 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN'
2016-01-11 15:15:45 5251 [Note] Shutting down plugin 'INNODB_SYS_FIELDS'
2016-01-11 15:15:45 5251 [Note] Shutting down plugin 'INNODB_SYS_COLUMNS'
2016-01-11 15:15:45 5251 [Note] Shutting down plugin 'INNODB_SYS_INDEXES'
2016-01-11 15:15:45 5251 [Note] Shutting down plugin 'INNODB_SYS_TABLESTATS'
2016-01-11 15:15:45 5251 [Note] Shutting down plugin 'INNODB_SYS_TABLES'
2016-01-11 15:15:45 5251 [Note] Shutting down plugin 'INNODB_FT_INDEX_TABLE'
2016-01-11 15:15:45 5251 [Note] Shutting down plugin 'INNODB_FT_INDEX_CACHE'
2016-01-11 15:15:45 5251 [Note] Shutting down plugin 'INNODB_FT_CONFIG'
2016-01-11 15:15:45 5251 [Note] Shutting down plugin 'INNODB_FT_BEING_DELETED'
2016-01-11 15:15:45 5251 [Note] Shutting down plugin 'INNODB_FT_DELETED'
2016-01-11 15:15:45 5251 [Note] Shutting down plugin 'INNODB_FT_DEFAULT_STOPWORD'
2016-01-11 15:15:45 5251 [Note] Shutting down plugin 'INNODB_METRICS'
2016-01-11 15:15:45 5251 [Note] Shutting down plugin 'INNODB_BUFFER_POOL_STATS'
2016-01-11 15:15:45 5251 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE_LRU'
2016-01-11 15:15:45 5251 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE'
2016-01-11 15:15:45 5251 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX_RESET'
2016-01-11 15:15:45 5251 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX'
2016-01-11 15:15:45 5251 [Note] Shutting down plugin 'INNODB_CMPMEM_RESET'
2016-01-11 15:15:45 5251 [Note] Shutting down plugin 'INNODB_CMPMEM'
2016-01-11 15:15:45 5251 [Note] Shutting down plugin 'INNODB_CMP_RESET'
2016-01-11 15:15:45 5251 [Note] Shutting down plugin 'INNODB_CMP'
2016-01-11 15:15:45 5251 [Note] Shutting down plugin 'INNODB_LOCK_WAITS'
2016-01-11 15:15:45 5251 [Note] Shutting down plugin 'INNODB_LOCKS'
2016-01-11 15:15:45 5251 [Note] Shutting down plugin 'INNODB_TRX'
2016-01-11 15:15:45 5251 [Note] Shutting down plugin 'XTRADB_RSEG'
2016-01-11 15:15:45 5251 [Note] Shutting down plugin 'XTRADB_INTERNAL_HASH_TABLES'
2016-01-11 15:15:45 5251 [Note] Shutting down plugin 'XTRADB_READ_VIEW'
2016-01-11 15:15:45 5251 [Note] Shutting down plugin 'InnoDB'
2016-01-11 15:15:45 5251 [Note] InnoDB: FTS optimize thread exiting.
2016-01-11 15:15:45 5251 [Note] InnoDB: Starting shutdown...
2016-01-11 15:15:47 5251 [Note] InnoDB: Shutdown completed; log sequence number 5680801480
2016-01-11 15:15:47 5251 [Note] Shutting down plugin 'MyISAM'
2016-01-11 15:15:47 5251 [Note] Shutting down plugin 'MRG_MYISAM'
2016-01-11 15:15:47 5251 [Note] Shutting down plugin 'MEMORY'
2016-01-11 15:15:47 5251 [Note] Shutting down plugin 'CSV'
2016-01-11 15:15:47 5251 [Note] Shutting down plugin 'sha256_password'
2016-01-11 15:15:47 5251 [Note] Shutting down plugin 'mysql_old_password'
2016-01-11 15:15:47 5251 [Note] Shutting down plugin 'mysql_native_password'
2016-01-11 15:15:47 5251 [Note] Shutting down plugin 'wsrep'
2016-01-11 15:15:47 5251 [Note] Shutting down plugin 'binlog'
2016-01-11 15:15:47 5251 [Note] /usr/sbin/mysqld: Shutdown complete

2016-01-11 15:15:49 0 [Note] /usr/sbin/mysqld (mysqld 5.6.27-76.0-56) starting as process 5413 ...
2016-01-11 15:15:49 5413 [Note] WSREP: Read nil XID from storage engines, skipping position init
2016-01-11 15:15:49 5413 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/libgalera_smm.so'
2016-01-11 15:15:49 5413 [Note] WSREP: wsrep_load(): Galera 3.13(rb4bea65) by Codership Oy <[email protected]> loaded successfully.
2016-01-11 15:15:49 5413 [Note] WSREP: CRC-32C: using hardware acceleration.
2016-01-11 15:15:49 5413 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1
2016-01-11 15:15:49 5413 [Note] WSREP: Passing config to GCS: base_dir = /srv/mysql/data/; base_host = 192.168.10.12; base_port = 4567; cert.log_conflicts = no; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /srv/mysql/data/; gcache.keep_pages_count = 0; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /srv/mysql/data//galera.cache; gcache.page_size = 128M; gcache.size = 128M; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc.checksum = false; pc.ignore_quor
2016-01-11 15:15:49 5413 [Note] WSREP: Service thread queue flushed.
2016-01-11 15:15:49 5413 [Note] WSREP: Assign initial position for certification: -1, protocol version: -1
2016-01-11 15:15:49 5413 [Note] WSREP: wsrep_sst_grab()
2016-01-11 15:15:49 5413 [Note] WSREP: Start replication
2016-01-11 15:15:49 5413 [Note] WSREP: Setting initial position to 00000000-0000-0000-0000-000000000000:-1
2016-01-11 15:15:49 5413 [Note] WSREP: protonet asio version 0
2016-01-11 15:15:49 5413 [Note] WSREP: Using CRC-32C for message checksums.
2016-01-11 15:15:49 5413 [Note] WSREP: backend: asio
2016-01-11 15:15:49 5413 [Note] WSREP: restore pc from disk successfully
2016-01-11 15:15:49 5413 [Note] WSREP: GMCast version 0
2016-01-11 15:15:49 5413 [Note] WSREP: (00000000, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567
2016-01-11 15:15:49 5413 [Note] WSREP: (00000000, 'tcp://0.0.0.0:4567') multicast: , ttl: 1
2016-01-11 15:15:49 5413 [ERROR] WSREP: failed to open gcomm backend connection: 131: invalid UUID: 00000000 (FATAL)
at gcomm/src/pc.cpp:PC():271
2016-01-11 15:15:49 5413 [ERROR] WSREP: gcs/src/gcs_core.cpp:gcs_core_open():208: Failed to open backend connection: -131 (State not recoverable)
2016-01-11 15:15:49 5413 [ERROR] WSREP: gcs/src/gcs.cpp:gcs_open():1379: Failed to open channel 'prestaserver_cluster' at 'gcomm://192.168.10.11,192.168.10.12,192.168.10.13': -131 (State not recoverable)
2016-01-11 15:15:49 5413 [ERROR] WSREP: gcs connect failed: State not recoverable
2016-01-11 15:15:49 5413 [ERROR] WSREP: wsrep::connect(gcomm://192.168.10.11,192.168.10.12,192.168.10.13) failed: 7
2016-01-11 15:15:49 5413 [ERROR] Aborting

2016-01-11 15:15:49 5413 [Note] WSREP: Service disconnected.
2016-01-11 15:15:50 5413 [Note] WSREP: Some threads may fail to exit.
2016-01-11 15:15:50 5413 [Note] Binlog end
2016-01-11 15:15:50 5413 [Note] /usr/sbin/mysqld: Shutdown complete

160111 15:15:50 mysqld_safe mysqld from pid file /srv/mysql/data/server2.pid ended
* The server quit without updating PID file (/srv/mysql/data/server2.pid).
* The server quit without updating PID file (/srv/mysql/data/server2.pid).

Tried to start mysql and forcing the donor to be the server1 but still the same issue

On the failed server:
-rw-rw---- 1 mysql mysql 104 Jan 11 15:29 grastate.dat
-rw-rw---- 1 mysql mysql 0 Jan 11 14:20 gvwstate.dat

cat grastate.dat
# GALERA saved state
version: 2.1
uuid: 00000000-0000-0000-0000-000000000000
seqno: -1
cert_index:


Any idea how to force the server to rsync and start?

Comments

  • PercUsePercUse Contributor Inactive User Role Beginner
    Tried again to restart the server specifying the donor to be server3


    Logs from the server2 (broken):
    16:24:17 mysqld_safe Starting mysqld daemon with databases from /srv/mysql/data
    160111 16:24:17 mysqld_safe WSREP: Running position recovery with --log_error='/srv/mysql/data/wsrep_recovery.aUOCr9' --pid-file='/srv/mysql/data/server2-recover.pid'
    2016-01-11 16:24:19 0 [Note] /usr/sbin/mysqld (mysqld 5.6.27-76.0-56) starting as process 16836 ...
    160111 16:24:55 mysqld_safe WSREP: Recovered position 51c9a3db-aeaa-11e3-a5e6-bffb1faf3524:1305553
    Log of wsrep recovery (--wsrep-recover):
    2016-01-11 16:24:19 16836 [Note] Plugin 'FEDERATED' is disabled.
    InnoDB: Warning: innodb_log_block_size has been changed from default value 512. (###EXPERIMENTAL### operation)
    InnoDB: The log block size is set to 4096.
    2016-01-11 16:24:19 16836 [Note] InnoDB: Using atomics to ref count buffer pool pages
    2016-01-11 16:24:19 16836 [Note] InnoDB: The InnoDB memory heap is disabled
    2016-01-11 16:24:19 16836 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
    2016-01-11 16:24:19 16836 [Note] InnoDB: Memory barrier is not used
    2016-01-11 16:24:19 16836 [Note] InnoDB: Compressed tables use zlib 1.2.3.4
    2016-01-11 16:24:19 16836 [Note] InnoDB: Using Linux native AIO
    2016-01-11 16:24:19 16836 [Note] InnoDB: Using CPU crc32 instructions
    2016-01-11 16:24:19 16836 [Note] InnoDB: Initializing buffer pool, size = 10.0G
    2016-01-11 16:24:19 16836 [Note] InnoDB: Completed initialization of buffer pool
    2016-01-11 16:24:19 16836 [Note] InnoDB: Highest supported file format is Barracuda.
    2016-01-11 16:24:52 16836 [Note] InnoDB: 128 rollback segment(s) are active.
    2016-01-11 16:24:52 16836 [Note] InnoDB: Waiting for purge to start
    2016-01-11 16:24:52 16836 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.27-76.0 started; log sequence number 5680802744
    2016-01-11 16:24:52 16836 [Warning] InnoDB: Setting thread 16865 nice to -10 failed, current nice 0, errno 13
    2016-01-11 16:24:52 16836 [Warning] InnoDB: Setting thread 16866 nice to -10 failed, current nice 0, errno 13
    2016-01-11 16:24:52 16836 [Warning] InnoDB: Skipping buffer pool dump/restore during wsrep recovery.
    2016-01-11 16:24:52 16836 [Note] RSA private key file not found: /srv/mysql/data//private_key.pem. Some authentication plugins will not work.
    2016-01-11 16:24:52 16836 [Note] RSA public key file not found: /srv/mysql/data//public_key.pem. Some authentication plugins will not work.
    2016-01-11 16:24:52 16836 [Note] Server hostname (bind-address): '192.168.10.12'; port: 3306
    2016-01-11 16:24:52 16836 [Note] - '192.168.10.12' resolves to '192.168.10.12';
    2016-01-11 16:24:52 16836 [Note] Server socket created on IP: '192.168.10.12'.
    2016-01-11 16:24:52 16836 [Note] WSREP: Recovered position: 51c9a3db-aeaa-11e3-a5e6-bffb1faf3524:1305553
    2016-01-11 16:24:52 16836 [Note] Binlog end
    2016-01-11 16:24:52 16836 [Note] Shutting down plugin 'partition'
    2016-01-11 16:24:52 16836 [Note] Shutting down plugin 'BLACKHOLE'
    2016-01-11 16:24:52 16836 [Note] Shutting down plugin 'ARCHIVE'
    2016-01-11 16:24:52 16836 [Note] Shutting down plugin 'PERFORMANCE_SCHEMA'
    2016-01-11 16:24:52 16836 [Note] Shutting down plugin 'INNODB_CHANGED_PAGES'
    2016-01-11 16:24:52 16836 [Note] Shutting down plugin 'INNODB_SYS_DATAFILES'
    2016-01-11 16:24:52 16836 [Note] Shutting down plugin 'INNODB_SYS_TABLESPACES'
    2016-01-11 16:24:52 16836 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN_COLS'
    2016-01-11 16:24:52 16836 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN'
    2016-01-11 16:24:52 16836 [Note] Shutting down plugin 'INNODB_SYS_FIELDS'
    2016-01-11 16:24:52 16836 [Note] Shutting down plugin 'INNODB_SYS_COLUMNS'
    2016-01-11 16:24:52 16836 [Note] Shutting down plugin 'INNODB_SYS_INDEXES'
    2016-01-11 16:24:52 16836 [Note] Shutting down plugin 'INNODB_SYS_TABLESTATS'
    2016-01-11 16:24:52 16836 [Note] Shutting down plugin 'INNODB_SYS_TABLES'
    2016-01-11 16:24:52 16836 [Note] Shutting down plugin 'INNODB_FT_INDEX_TABLE'
    2016-01-11 16:24:52 16836 [Note] Shutting down plugin 'INNODB_FT_INDEX_CACHE'
    2016-01-11 16:24:52 16836 [Note] Shutting down plugin 'INNODB_FT_CONFIG'
    2016-01-11 16:24:52 16836 [Note] Shutting down plugin 'INNODB_FT_BEING_DELETED'
    2016-01-11 16:24:52 16836 [Note] Shutting down plugin 'INNODB_FT_DELETED'
    2016-01-11 16:24:52 16836 [Note] Shutting down plugin 'INNODB_FT_DEFAULT_STOPWORD'
    2016-01-11 16:24:52 16836 [Note] Shutting down plugin 'INNODB_METRICS'
    2016-01-11 16:24:52 16836 [Note] Shutting down plugin 'INNODB_BUFFER_POOL_STATS'
    2016-01-11 16:24:52 16836 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE_LRU'
    2016-01-11 16:24:52 16836 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE'
    2016-01-11 16:24:52 16836 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX_RESET'
    2016-01-11 16:24:52 16836 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX'
    2016-01-11 16:24:52 16836 [Note] Shutting down plugin 'INNODB_CMPMEM_RESET'
    2016-01-11 16:24:52 16836 [Note] Shutting down plugin 'INNODB_CMPMEM'
    2016-01-11 16:24:52 16836 [Note] Shutting down plugin 'INNODB_CMP_RESET'
    2016-01-11 16:24:52 16836 [Note] Shutting down plugin 'INNODB_CMP'
    2016-01-11 16:24:52 16836 [Note] Shutting down plugin 'INNODB_LOCK_WAITS'
    2016-01-11 16:24:52 16836 [Note] Shutting down plugin 'INNODB_LOCKS'
    2016-01-11 16:24:52 16836 [Note] Shutting down plugin 'INNODB_TRX'
    2016-01-11 16:24:52 16836 [Note] Shutting down plugin 'XTRADB_RSEG'
    2016-01-11 16:24:52 16836 [Note] Shutting down plugin 'XTRADB_INTERNAL_HASH_TABLES'
    2016-01-11 16:24:52 16836 [Note] Shutting down plugin 'XTRADB_READ_VIEW'
    2016-01-11 16:24:52 16836 [Note] Shutting down plugin 'InnoDB'
    2016-01-11 16:24:52 16836 [Note] InnoDB: FTS optimize thread exiting.
    2016-01-11 16:24:52 16836 [Note] InnoDB: Starting shutdown...
    2016-01-11 16:24:55 16836 [Note] InnoDB: Shutdown completed; log sequence number 5680802754
    2016-01-11 16:24:55 16836 [Note] Shutting down plugin 'MyISAM'
    2016-01-11 16:24:55 16836 [Note] Shutting down plugin 'MRG_MYISAM'
    2016-01-11 16:24:55 16836 [Note] Shutting down plugin 'MEMORY'
    2016-01-11 16:24:55 16836 [Note] Shutting down plugin 'CSV'
    2016-01-11 16:24:55 16836 [Note] Shutting down plugin 'sha256_password'
    2016-01-11 16:24:55 16836 [Note] Shutting down plugin 'mysql_old_password'
    2016-01-11 16:24:55 16836 [Note] Shutting down plugin 'mysql_native_password'
    2016-01-11 16:24:55 16836 [Note] Shutting down plugin 'wsrep'
    2016-01-11 16:24:55 16836 [Note] Shutting down plugin 'binlog'
    2016-01-11 16:24:55 16836 [Note] /usr/sbin/mysqld: Shutdown complete

    2016-01-11 16:24:56 0 [Note] /usr/sbin/mysqld (mysqld 5.6.27-76.0-56) starting as process 16880 ...
    2016-01-11 16:24:56 16880 [Note] WSREP: Read nil XID from storage engines, skipping position init
    2016-01-11 16:24:56 16880 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/libgalera_smm.so'
    2016-01-11 16:24:56 16880 [Note] WSREP: wsrep_load(): Galera 3.13(rb4bea65) by Codership Oy <[email protected]> loaded successfully.
    2016-01-11 16:24:56 16880 [Note] WSREP: CRC-32C: using hardware acceleration.
    2016-01-11 16:24:56 16880 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1
    2016-01-11 16:24:56 16880 [Note] WSREP: Passing config to GCS: base_dir = /srv/mysql/data/; base_host = 192.168.10.12; base_port = 4567; cert.log_conflicts = no; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /srv/mysql/data/; gcache.keep_pages_count = 0; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /srv/mysql/data//galera.cache; gcache.page_size = 128M; gcache.size = 128M; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc.checksum = false; pc.ignore_quor
    2016-01-11 16:24:56 16880 [Note] WSREP: Service thread queue flushed.
    2016-01-11 16:24:56 16880 [Note] WSREP: Assign initial position for certification: -1, protocol version: -1
    2016-01-11 16:24:56 16880 [Note] WSREP: wsrep_sst_grab()
    2016-01-11 16:24:56 16880 [Note] WSREP: Start replication
    2016-01-11 16:24:56 16880 [Note] WSREP: Setting initial position to 00000000-0000-0000-0000-000000000000:-1
    2016-01-11 16:24:56 16880 [Note] WSREP: protonet asio version 0
    2016-01-11 16:24:56 16880 [Note] WSREP: Using CRC-32C for message checksums.
    2016-01-11 16:24:56 16880 [Note] WSREP: backend: asio
    2016-01-11 16:24:56 16880 [Warning] WSREP: access file(/srv/mysql/data//gvwstate.dat) failed(No such file or directory)
    2016-01-11 16:24:56 16880 [Note] WSREP: restore pc from disk failed
    2016-01-11 16:24:56 16880 [Note] WSREP: GMCast version 0
    2016-01-11 16:24:56 16880 [Note] WSREP: (789476f2, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567
    2016-01-11 16:24:56 16880 [Note] WSREP: (789476f2, 'tcp://0.0.0.0:4567') multicast: , ttl: 1
    2016-01-11 16:24:56 16880 [Note] WSREP: EVS version 0
    2016-01-11 16:24:56 16880 [Note] WSREP: gcomm: connecting to group 'prestaserver_cluster', peer '192.168.10.11:,192.168.10.12:,192.168.10.13:'
    2016-01-11 16:24:56 16880 [Warning] WSREP: (789476f2, 'tcp://0.0.0.0:4567') address 'tcp://192.168.10.12:4567' points to own listening address, blacklisting
    2016-01-11 16:24:56 16880 [Note] WSREP: (789476f2, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers:
    2016-01-11 16:24:57 16880 [Note] WSREP: declaring 0ee12c0b at tcp://192.168.10.13:4567 stable
    2016-01-11 16:24:57 16880 [Note] WSREP: declaring dae17fee at tcp://192.168.10.11:4567 stable
    2016-01-11 16:24:57 16880 [Note] WSREP: Node 0ee12c0b state prim
    2016-01-11 16:24:57 16880 [Note] WSREP: view(view_id(PRIM,0ee12c0b,10) memb {
    0ee12c0b,0
    789476f2,0
    dae17fee,0
    } joined {
    } left {
    } partitioned {
    })
    2016-01-11 16:24:57 16880 [Note] WSREP: save pc into disk
    2016-01-11 16:24:57 16880 [Note] WSREP: gcomm: connected
    2016-01-11 16:24:57 16880 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636
    2016-01-11 16:24:57 16880 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)
    2016-01-11 16:24:57 16880 [Note] WSREP: Opened channel 'prestaserver_cluster'
    2016-01-11 16:24:57 16880 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 3
    2016-01-11 16:24:57 16880 [Note] WSREP: Waiting for SST to complete.
    2016-01-11 16:24:57 16880 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
    2016-01-11 16:24:57 16880 [Note] WSREP: STATE EXCHANGE: sent state msg: 78e1882a-b877-11e5-a6cd-0ea397aeb4b4
    2016-01-11 16:24:57 16880 [Note] WSREP: STATE EXCHANGE: got state msg: 78e1882a-b877-11e5-a6cd-0ea397aeb4b4 from 0 (server3in)
    2016-01-11 16:24:57 16880 [Note] WSREP: STATE EXCHANGE: got state msg: 78e1882a-b877-11e5-a6cd-0ea397aeb4b4 from 2 (server1in)
    2016-01-11 16:24:57 16880 [Note] WSREP: STATE EXCHANGE: got state msg: 78e1882a-b877-11e5-a6cd-0ea397aeb4b4 from 1 (server2in)
    2016-01-11 16:24:57 16880 [Note] WSREP: Quorum results:
    version = 3,
    component = PRIMARY,
    conf_id = 8,
    members = 2/3 (joined/total),
    act_id = 1309360,
    last_appl. = -1,
    protocols = 0/7/3 (gcs/repl/appl),
    group UUID = 51c9a3db-aeaa-11e3-a5e6-bffb1faf3524
    2016-01-11 16:24:57 16880 [Note] WSREP: Flow-control interval: [28, 28]
    2016-01-11 16:24:57 16880 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 1309360)
    2016-01-11 16:24:57 16880 [Note] WSREP: State transfer required:
    Group state: 51c9a3db-aeaa-11e3-a5e6-bffb1faf3524:1309360
    Local state: 00000000-0000-0000-0000-000000000000:-1
    2016-01-11 16:24:57 16880 [Note] WSREP: New cluster view: global state: 51c9a3db-aeaa-11e3-a5e6-bffb1faf3524:1309360, view# 9: Primary, number of nodes: 3, my index: 1, protocol version 3
    2016-01-11 16:24:57 16880 [Warning] WSREP: Gap in state sequence. Need state transfer.
    2016-01-11 16:24:57 16880 [Note] WSREP: Running: 'wsrep_sst_xtrabackup-v2 --role 'joiner' --address '192.168.10.12' --datadir '/srv/mysql/data/' --defaults-file '/etc/mysql/my.cnf' --defaults-group-suffix '' --parent '16880' '' '
    WSREP_SST: [INFO] Streaming with xbstream (20160111 16:24:57.947)
    WSREP_SST: [INFO] Using socat as streamer (20160111 16:24:57.948)
    WSREP_SST: [INFO] Stale sst_in_progress file: /srv/mysql/data//sst_in_progress (20160111 16:24:57.952)
    WSREP_SST: [INFO] Evaluating timeout -k 110 100 socat -u TCP-LISTEN:4444,reuseaddr stdio | xbstream -x; RC=( ${PIPESTATUS[@]} ) (20160111 16:24:58.061)
    2016-01-11 16:24:58 16880 [Note] WSREP: Prepared SST request: xtrabackup-v2|192.168.10.12:4444/xtrabackup_sst//1
    2016-01-11 16:24:58 16880 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
    2016-01-11 16:24:58 16880 [Note] WSREP: REPL Protocols: 7 (3, 2)
    2016-01-11 16:24:58 16880 [Note] WSREP: Service thread queue flushed.
    2016-01-11 16:24:58 16880 [Note] WSREP: Assign initial position for certification: 1309360, protocol version: 3
    2016-01-11 16:24:58 16880 [Note] WSREP: Service thread queue flushed.
    2016-01-11 16:24:58 16880 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (51c9a3db-aeaa-11e3-a5e6-bffb1faf3524): 1 (Operation not permitted)
    at galera/src/replicator_str.cpp:prepare_for_IST():489. IST will be unavailable.
    2016-01-11 16:24:58 16880 [Note] WSREP: Member 1.0 (server2in) requested state transfer from '*any*'. Selected 0.0 (server3in)(SYNCED) as donor.
    2016-01-11 16:24:58 16880 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 1309360)
    2016-01-11 16:24:58 16880 [Note] WSREP: Requesting state transfer: success, donor: 0
    WSREP_SST: [ERROR] Parent mysqld process (PID:16880) terminated unexpectedly. (20160111 16:24:58.443)
    WSREP_SST: [ERROR] Cleanup after exit with status:32 (20160111 16:24:58.446)
    2016-01-11 16:24:58 16880 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'joiner' --address '192.168.10.12' --datadir '/srv/mysql/data/' --defaults-file '/etc/mysql/my.cnf' --defaults-group-suffix '' --parent '16880' '' : 32 (Broken pipe)
    2016-01-11 16:24:58 16880 [ERROR] WSREP: Failed to read uuid:seqno from joiner script.
    2016-01-11 16:24:58 16880 [ERROR] WSREP: SST script aborted with error 32 (Broken pipe)
    2016-01-11 16:24:58 16880 [ERROR] WSREP: SST failed: 32 (Broken pipe)
    2016-01-11 16:24:58 16880 [ERROR] Aborting

    2016-01-11 16:25:00 16880 [Note] WSREP: (789476f2, 'tcp://0.0.0.0:4567') turning message relay requesting off
    2016-01-11 16:25:00 16880 [Note] WSREP: Closing send monitor...
    2016-01-11 16:25:00 16880 [Note] WSREP: Closed send monitor.
    2016-01-11 16:25:00 16880 [Note] WSREP: gcomm: terminating thread
    2016-01-11 16:25:00 16880 [Note] WSREP: gcomm: joining thread
    2016-01-11 16:25:00 16880 [Note] WSREP: gcomm: closing backend
    2016-01-11 16:25:00 16880 [Note] WSREP: view(view_id(NON_PRIM,0ee12c0b,10) memb {
    789476f2,0
    } joined {
    } left {
    } partitioned {
    0ee12c0b,0
    dae17fee,0
    })
    2016-01-11 16:25:00 16880 [Note] WSREP: view((empty))
    2016-01-11 16:25:00 16880 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
    2016-01-11 16:25:00 16880 [Note] WSREP: gcomm: closed
    2016-01-11 16:25:00 16880 [Note] WSREP: Flow-control interval: [16, 16]
    2016-01-11 16:25:00 16880 [Note] WSREP: Received NON-PRIMARY.
    2016-01-11 16:25:00 16880 [Note] WSREP: Shifting JOINER -> OPEN (TO: 1309360)
    2016-01-11 16:25:00 16880 [Note] WSREP: Received self-leave message.
    2016-01-11 16:25:00 16880 [Note] WSREP: Flow-control interval: [0, 0]
    2016-01-11 16:25:00 16880 [Note] WSREP: Received SELF-LEAVE. Closing connection.
    2016-01-11 16:25:00 16880 [Note] WSREP: Shifting OPEN -> CLOSED (TO: 1309360)
    2016-01-11 16:25:00 16880 [Note] WSREP: RECV thread exiting 0: Success
    2016-01-11 16:25:00 16880 [Note] WSREP: recv_thread() joined.
    2016-01-11 16:25:00 16880 [Note] WSREP: Closing replication queue.
    2016-01-11 16:25:00 16880 [Note] WSREP: Closing slave action queue.
    2016-01-11 16:25:00 16880 [Note] WSREP: Service disconnected.
    2016-01-11 16:25:00 16880 [Note] WSREP: rollbacker thread exiting
    2016-01-11 16:25:01 16880 [Note] WSREP: Some threads may fail to exit.
    2016-01-11 16:25:01 16880 [Note] Binlog end
    2016-01-11 16:25:01 16880 [Note] /usr/sbin/mysqld: Shutdown complete

    Error in my_thread_global_end(): 1 threads didn't exit
    160111 16:25:06 mysqld_safe mysqld from pid file /srv/mysql/data/server2.pid ended

    Logs from server3 (donor):
    2016-01-11 16:24:56 6540 [Note] WSREP: (0ee12c0b, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers:
    2016-01-11 16:24:57 6540 [Note] WSREP: declaring 789476f2 at tcp://192.168.10.12:4567 stable
    2016-01-11 16:24:57 6540 [Note] WSREP: declaring dae17fee at tcp://192.168.10.11:4567 stable
    2016-01-11 16:24:57 6540 [Note] WSREP: Node 0ee12c0b state prim
    2016-01-11 16:24:57 6540 [Note] WSREP: view(view_id(PRIM,0ee12c0b,10) memb {
    0ee12c0b,0
    789476f2,0
    dae17fee,0
    } joined {
    } left {
    } partitioned {
    })
    2016-01-11 16:24:57 6540 [Note] WSREP: save pc into disk
    2016-01-11 16:24:57 6540 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 3
    2016-01-11 16:24:57 6540 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 78e1882a-b877-11e5-a6cd-0ea397aeb4b4
    2016-01-11 16:24:57 6540 [Note] WSREP: STATE EXCHANGE: sent state msg: 78e1882a-b877-11e5-a6cd-0ea397aeb4b4
    2016-01-11 16:24:57 6540 [Note] WSREP: STATE EXCHANGE: got state msg: 78e1882a-b877-11e5-a6cd-0ea397aeb4b4 from 0 (server3in)
    2016-01-11 16:24:57 6540 [Note] WSREP: STATE EXCHANGE: got state msg: 78e1882a-b877-11e5-a6cd-0ea397aeb4b4 from 2 (server1in)
    2016-01-11 16:24:57 6540 [Note] WSREP: STATE EXCHANGE: got state msg: 78e1882a-b877-11e5-a6cd-0ea397aeb4b4 from 1 (server2in)
    2016-01-11 16:24:57 6540 [Note] WSREP: Quorum results:
    version = 3,
    component = PRIMARY,
    conf_id = 8,
    members = 2/3 (joined/total),
    act_id = 1309360,
    last_appl. = 1309329,
    protocols = 0/7/3 (gcs/repl/appl),
    group UUID = 51c9a3db-aeaa-11e3-a5e6-bffb1faf3524
    2016-01-11 16:24:57 6540 [Note] WSREP: Flow-control interval: [28, 28]
    2016-01-11 16:24:57 6540 [Note] WSREP: New cluster view: global state: 51c9a3db-aeaa-11e3-a5e6-bffb1faf3524:1309360, view# 9: Primary, number of nodes: 3, my index: 0, protocol version 3
    2016-01-11 16:24:57 6540 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
    2016-01-11 16:24:57 6540 [Note] WSREP: REPL Protocols: 7 (3, 2)
    2016-01-11 16:24:57 6540 [Note] WSREP: Service thread queue flushed.
    2016-01-11 16:24:57 6540 [Note] WSREP: Assign initial position for certification: 1309360, protocol version: 3
    2016-01-11 16:24:57 6540 [Note] WSREP: Service thread queue flushed.
    2016-01-11 16:24:58 6540 [Note] WSREP: Member 1.0 (server2in) requested state transfer from '*any*'. Selected 0.0 (server3in)(SYNCED) as donor.
    2016-01-11 16:24:58 6540 [Note] WSREP: Shifting SYNCED -> DONOR/DESYNCED (TO: 1309360)
    2016-01-11 16:24:58 6540 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
    2016-01-11 16:24:58 6540 [Note] WSREP: Running: 'wsrep_sst_xtrabackup-v2 --role 'donor' --address '192.168.10.12:4444/xtrabackup_sst//1' --socket '/var/run/mysqld/mysqld.sock' --datadir '/srv/mysql/data/' --defaults-file '/etc/mysql/my.cnf' --defaults-group-suffix '' '' --gtid '51c9a3db-aeaa-11e3-a5e6-bffb1faf3524:1309360''
    2016-01-11 16:24:58 6540 [Note] WSREP: sst_donor_thread signaled with 0
    WSREP_SST: [INFO] Streaming with xbstream (20160111 16:24:58.364)
    WSREP_SST: [INFO] Using socat as streamer (20160111 16:24:58.367)
    WSREP_SST: [INFO] Using /tmp/tmp.oPsbIOa4Wn as xtrabackup temporary directory (20160111 16:24:58.384)
    WSREP_SST: [INFO] Using /tmp/tmp.Hek7hwKeX2 as innobackupex temporary directory (20160111 16:24:58.387)
    WSREP_SST: [INFO] Streaming GTID file before SST (20160111 16:24:58.393)
    WSREP_SST: [INFO] Evaluating xbstream -c ${INFO_FILE} | socat -u stdio TCP:192.168.10.12:4444; RC=( ${PIPESTATUS[@]} ) (20160111 16:24:58.399)
    WSREP_SST: [INFO] Sleeping before data transfer for SST (20160111 16:24:58.407)
    2016-01-11 16:25:00 6540 [Note] WSREP: (0ee12c0b, 'tcp://0.0.0.0:4567') turning message relay requesting off
    2016-01-11 16:25:00 6540 [Note] WSREP: declaring dae17fee at tcp://192.168.10.11:4567 stable
    2016-01-11 16:25:00 6540 [Note] WSREP: forgetting 789476f2 (tcp://192.168.10.12:4567)
    2016-01-11 16:25:00 6540 [Note] WSREP: Node 0ee12c0b state prim
    2016-01-11 16:25:00 6540 [Note] WSREP: view(view_id(PRIM,0ee12c0b,11) memb {
    0ee12c0b,0
    dae17fee,0
    } joined {
    } left {
    } partitioned {
    789476f2,0
    })
    2016-01-11 16:25:00 6540 [Note] WSREP: save pc into disk
    2016-01-11 16:25:00 6540 [Note] WSREP: forgetting 789476f2 (tcp://192.168.10.12:4567)
    2016-01-11 16:25:00 6540 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 2
    2016-01-11 16:25:00 6540 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 7abdc036-b877-11e5-88e6-9a7ccc4b0bda
    2016-01-11 16:25:00 6540 [Note] WSREP: STATE EXCHANGE: sent state msg: 7abdc036-b877-11e5-88e6-9a7ccc4b0bda
    2016-01-11 16:25:00 6540 [Note] WSREP: STATE EXCHANGE: got state msg: 7abdc036-b877-11e5-88e6-9a7ccc4b0bda from 0 (server3in)
    2016-01-11 16:25:00 6540 [Note] WSREP: STATE EXCHANGE: got state msg: 7abdc036-b877-11e5-88e6-9a7ccc4b0bda from 1 (server1in)
    2016-01-11 16:25:00 6540 [Note] WSREP: Quorum results:
    version = 3,
    component = PRIMARY,
    conf_id = 9,
    members = 2/2 (joined/total),
    act_id = 1309360,
    last_appl. = 1309329,
    protocols = 0/7/3 (gcs/repl/appl),
    group UUID = 51c9a3db-aeaa-11e3-a5e6-bffb1faf3524
    2016-01-11 16:25:00 6540 [Note] WSREP: Flow-control interval: [23, 23]
    2016-01-11 16:25:00 6540 [Note] WSREP: New cluster view: global state: 51c9a3db-aeaa-11e3-a5e6-bffb1faf3524:1309360, view# 10: Primary, number of nodes: 2, my index: 0, protocol version 3
    2016-01-11 16:25:00 6540 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
    2016-01-11 16:25:00 6540 [Note] WSREP: REPL Protocols: 7 (3, 2)
    2016-01-11 16:25:00 6540 [Note] WSREP: Service thread queue flushed.
    2016-01-11 16:25:00 6540 [Note] WSREP: Assign initial position for certification: 1309360, protocol version: 3
    2016-01-11 16:25:00 6540 [Note] WSREP: Service thread queue flushed.
    2016-01-11 16:25:05 6540 [Note] WSREP: cleaning up 789476f2 (tcp://192.168.10.12:4567)
    WSREP_SST: [INFO] Streaming the backup to joiner at 192.168.10.12 4444 (20160111 16:25:08.412)
    WSREP_SST: [INFO] Evaluating innobackupex --defaults-file=/etc/mysql/my.cnf --defaults-group=mysqld --no-version-check $tmpopts $INNOEXTRA --galera-info --stream=$sfmt $itmpdir 2>${DATA}/innobackup.backup.log | socat -u stdio TCP:192.168.10.12:4444; RC=( ${PIPESTATUS[@]} ) (20160111 16:25:08.416)
    2016/01/11 16:25:08 socat[27142] E connect(3, AF=2 192.168.10.12:4444, 16): Connection refused
    WSREP_SST: [ERROR] innobackupex finished with error: 1. Check /srv/mysql/data//innobackup.backup.log (20160111 16:25:09.574)
    WSREP_SST: [ERROR] Cleanup after exit with status:22 (20160111 16:25:09.578)
    WSREP_SST: [INFO] Cleaning up temporary directories (20160111 16:25:09.582)
    2016-01-11 16:25:09 6540 [ERROR] WSREP: Failed to read from: wsrep_sst_xtrabackup-v2 --role 'donor' --address '192.168.10.12:4444/xtrabackup_sst//1' --socket '/var/run/mysqld/mysqld.sock' --datadir '/srv/mysql/data/' --defaults-file '/etc/mysql/my.cnf' --defaults-group-suffix '' '' --gtid '51c9a3db-aeaa-11e3-a5e6-bffb1faf3524:1309360'
    2016-01-11 16:25:09 6540 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'donor' --address '192.168.10.12:4444/xtrabackup_sst//1' --socket '/var/run/mysqld/mysqld.sock' --datadir '/srv/mysql/data/' --defaults-file '/etc/mysql/my.cnf' --defaults-group-suffix '' '' --gtid '51c9a3db-aeaa-11e3-a5e6-bffb1faf3524:1309360': 22 (Invalid argument)
    2016-01-11 16:25:09 6540 [ERROR] WSREP: Command did not run: wsrep_sst_xtrabackup-v2 --role 'donor' --address '192.168.10.12:4444/xtrabackup_sst//1' --socket '/var/run/mysqld/mysqld.sock' --datadir '/srv/mysql/data/' --defaults-file '/etc/mysql/my.cnf' --defaults-group-suffix '' '' --gtid '51c9a3db-aeaa-11e3-a5e6-bffb1faf3524:1309360'
    2016-01-11 16:25:09 6540 [Warning] WSREP: Could not find peer: 789476f2-b877-11e5-b548-fee1e647e6b6
    2016-01-11 16:25:09 6540 [Warning] WSREP: 0.0 (server3in): State transfer to -1.-1 (left the group) failed: -22 (Invalid argument)
    2016-01-11 16:25:09 6540 [Note] WSREP: Shifting DONOR/DESYNCED -> JOINED (TO: 1309360)
    2016-01-11 16:25:09 6540 [Note] WSREP: Member 0.0 (server3in) synced with group.
    2016-01-11 16:25:09 6540 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 1309360)
    2016-01-11 16:25:09 6540 [Note] WSREP: Synchronized with group, ready for connections
    2016-01-11 16:25:09 6540 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
  • PercUsePercUse Contributor Inactive User Role Beginner
    Any help or idea will be welcome. ;)

    I tried to force SST sync but it fail each time (open file limit ?)

    Seems to be hard to recover this server :s

    2016-01-12 15:14:13 28030 [Warning] InnoDB: Cannot open table web567/ps_smarty_lazy_cache from the internal data dictionary of InnoDB though the .frm file for the table exists. See http://dev.mysql.com/doc/refman/5.6/...eshooting.html for how you can resolve the problem.
    2016-01-12 15:14:13 28030 [Warning] WSREP: BF applier failed to open_and_lock_tables: 1146, fatal: 0 wsrep = (exec_mode: 1 conflict_state: 0 seqno: 1320051)
    2016-01-12 15:14:13 28030 [ERROR] Slave SQL: Error executing row event: 'Table 'web567.ps_smarty_lazy_cache' doesn't exist', Error_code: 1146
    2016-01-12 15:14:13 28030 [Warning] WSREP: RBR event 3 Write_rows apply warning: 1146, 1320051
    2016-01-12 15:14:13 28030 [ERROR] WSREP: Failed to apply trx: source: e8566194-b935-11e5-94d5-139a15c6d8c2 version: 3 local: 0 state: APPLYING flags: 1 conn_id: 738 trx_id: 7013954 seqnos (l: 5, g: 1320051, s: 1320050, d: 1320050, ts: 2987010053581795)
    2016-01-12 15:14:13 28030 [ERROR] WSREP: Failed to apply trx 1320051 4 times
    2016-01-12 15:14:13 28030 [ERROR] WSREP: Node consistency compromized, aborting...
    2016-01-12 15:14:13 28030 [Note] WSREP: Closing send monitor...
    2016-01-12 15:14:13 28030 [Note] WSREP: Closed send monitor.
    2016-01-12 15:14:13 28030 [Note] WSREP: gcomm: terminating thread
    2016-01-12 15:14:13 28030 [Note] WSREP: gcomm: joining thread
    2016-01-12 15:14:13 28030 [Note] WSREP: gcomm: closing backend
    2016-01-12 15:14:13 28030 [Note] WSREP: view(view_id(NON_PRIM,4eb463f5,33) memb {
    94582432,0
    } joined {
    } left {
    } partitioned {
    4eb463f5,0
    e8566194,0
    })
    2016-01-12 15:14:13 28030 [Note] WSREP: view((empty))
    2016-01-12 15:14:13 28030 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
    2016-01-12 15:14:13 28030 [Note] WSREP: gcomm: closed
    2016-01-12 15:14:13 28030 [Note] WSREP: Flow-control interval: [16, 16]
    2016-01-12 15:14:13 28030 [Note] WSREP: Received NON-PRIMARY.
    2016-01-12 15:14:13 28030 [Note] WSREP: Shifting SYNCED -> OPEN (TO: 1320051)
    2016-01-12 15:14:13 28030 [Note] WSREP: Received self-leave message.
    2016-01-12 15:14:13 28030 [Note] WSREP: Flow-control interval: [0, 0]
    2016-01-12 15:14:13 28030 [Note] WSREP: Received SELF-LEAVE. Closing connection.
    2016-01-12 15:14:13 28030 [Note] WSREP: Shifting OPEN -> CLOSED (TO: 1320051)
    2016-01-12 15:14:13 28030 [Note] WSREP: RECV thread exiting 0: Success
    2016-01-12 15:14:13 28030 [Note] WSREP: recv_thread() joined.
    2016-01-12 15:14:13 28030 [Note] WSREP: Closing replication queue.
    2016-01-12 15:14:13 28030 [Note] WSREP: Closing slave action queue.
    2016-01-12 15:14:13 28030 [Note] WSREP: /usr/sbin/mysqld: Terminated.
    Aborted (core dumped)
    160112 15:14:13 mysqld_safe Number of processes running now: 0
    160112 15:14:13 mysqld_safe WSREP: not restarting wsrep node automatically


    And It seems that we have this error with all DB on the affected node


    The affected node got kicked and die when triying to restart it:
    [email protected]:~# tail -f /srv/mysql/data/wsrep_recovery.gKOov7
    2016-01-12 15:17:30 29954 [Note] InnoDB: Compressed tables use zlib 1.2.3.4
    2016-01-12 15:17:30 29954 [Note] InnoDB: Using Linux native AIO
    2016-01-12 15:17:30 29954 [Note] InnoDB: Using CPU crc32 instructions
    2016-01-12 15:17:30 29954 [Note] InnoDB: Initializing buffer pool, size = 10.0G
    2016-01-12 15:17:30 29954 [Note] InnoDB: Completed initialization of buffer pool
    2016-01-12 15:17:31 29954 [Note] InnoDB: Highest supported file format is Barracuda.
    2016-01-12 15:17:31 29954 [Note] InnoDB: The log sequence numbers 94245707 and 94245707 in ibdata files do not match the log sequence number 94245871 in the ib_logfiles!
    2016-01-12 15:17:31 29954 [Note] InnoDB: Database was not shutdown normally!
    2016-01-12 15:17:31 29954 [Note] InnoDB: Starting crash recovery.
    2016-01-12 15:17:31 29954 [Note] InnoDB: Reading tablespace information from the .ibd files...
    2016-01-12 15:18:46 29954 [Note] InnoDB: Restoring possible half-written data pages
    2016-01-12 15:18:46 29954 [Note] InnoDB: from the doublewrite buffer...
    2016-01-12 15:18:46 29954 [Note] InnoDB: 128 rollback segment(s) are active.
    2016-01-12 15:18:46 29954 [Note] InnoDB: Waiting for purge to start
    2016-01-12 15:18:46 29954 [Warning] InnoDB: Setting thread 30372 nice to -10 failed, current nice 0, errno 13
    2016-01-12 15:18:46 29954 [Warning] InnoDB: Setting thread 30373 nice to -10 failed, current nice 0, errno 13
    2016-01-12 15:18:46 29954 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.27-76.0 started; log sequence number 94245871
    2016-01-12 15:18:46 29954 [Warning] InnoDB: Skipping buffer pool dump/restore during wsrep recovery.
    2016-01-12 15:18:46 29954 [Note] RSA private key file not found: /srv/mysql/data//private_key.pem. Some authentication plugins will not work.
    2016-01-12 15:18:46 29954 [Note] RSA public key file not found: /srv/mysql/data//public_key.pem. Some authentication plugins will not work.
    2016-01-12 15:18:46 29954 [Note] Server hostname (bind-address): '192.168.10.12'; port: 3306
    2016-01-12 15:18:46 29954 [Note] - '192.168.10.12' resolves to '192.168.10.12';
    2016-01-12 15:18:46 29954 [Note] Server socket created on IP: '192.168.10.12'.
    2016-01-12 15:18:46 29954 [Note] WSREP: Recovered position: 51c9a3db-aeaa-11e3-a5e6-bffb1faf3524:1320050
    2016-01-12 15:18:46 29954 [Note] Binlog end
    2016-01-12 15:18:46 29954 [Note] Shutting down plugin 'partition'
    2016-01-12 15:18:46 29954 [Note] Shutting down plugin 'BLACKHOLE'
    2016-01-12 15:18:46 29954 [Note] Shutting down plugin 'ARCHIVE'
    2016-01-12 15:18:46 29954 [Note] Shutting down plugin 'PERFORMANCE_SCHEMA'
    2016-01-12 15:18:46 29954 [Note] Shutting down plugin 'INNODB_CHANGED_PAGES'
    2016-01-12 15:18:46 29954 [Note] Shutting down plugin 'INNODB_SYS_DATAFILES'
    2016-01-12 15:18:46 29954 [Note] Shutting down plugin 'INNODB_SYS_TABLESPACES'
    2016-01-12 15:18:46 29954 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN_COLS'
    2016-01-12 15:18:46 29954 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN'
    2016-01-12 15:18:46 29954 [Note] Shutting down plugin 'INNODB_SYS_FIELDS'
    2016-01-12 15:18:46 29954 [Note] Shutting down plugin 'INNODB_SYS_COLUMNS'
    2016-01-12 15:18:46 29954 [Note] Shutting down plugin 'INNODB_SYS_INDEXES'
    2016-01-12 15:18:46 29954 [Note] Shutting down plugin 'INNODB_SYS_TABLESTATS'
    2016-01-12 15:18:46 29954 [Note] Shutting down plugin 'INNODB_SYS_TABLES'
    2016-01-12 15:18:46 29954 [Note] Shutting down plugin 'INNODB_FT_INDEX_TABLE'
    2016-01-12 15:18:46 29954 [Note] Shutting down plugin 'INNODB_FT_INDEX_CACHE'
    2016-01-12 15:18:46 29954 [Note] Shutting down plugin 'INNODB_FT_CONFIG'
    2016-01-12 15:18:46 29954 [Note] Shutting down plugin 'INNODB_FT_BEING_DELETED'
    2016-01-12 15:18:46 29954 [Note] Shutting down plugin 'INNODB_FT_DELETED'
    2016-01-12 15:18:46 29954 [Note] Shutting down plugin 'INNODB_FT_DEFAULT_STOPWORD'
    2016-01-12 15:18:46 29954 [Note] Shutting down plugin 'INNODB_METRICS'
    2016-01-12 15:18:46 29954 [Note] Shutting down plugin 'INNODB_BUFFER_POOL_STATS'
    2016-01-12 15:18:46 29954 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE_LRU'
    2016-01-12 15:18:46 29954 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE'
    2016-01-12 15:18:46 29954 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX_RESET'
    2016-01-12 15:18:46 29954 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX'
    2016-01-12 15:18:46 29954 [Note] Shutting down plugin 'INNODB_CMPMEM_RESET'
    2016-01-12 15:18:46 29954 [Note] Shutting down plugin 'INNODB_CMPMEM'
    2016-01-12 15:18:46 29954 [Note] Shutting down plugin 'INNODB_CMP_RESET'
    2016-01-12 15:18:46 29954 [Note] Shutting down plugin 'INNODB_CMP'
    2016-01-12 15:18:46 29954 [Note] Shutting down plugin 'INNODB_LOCK_WAITS'
    2016-01-12 15:18:46 29954 [Note] Shutting down plugin 'INNODB_LOCKS'
    2016-01-12 15:18:46 29954 [Note] Shutting down plugin 'INNODB_TRX'
    2016-01-12 15:18:46 29954 [Note] Shutting down plugin 'XTRADB_RSEG'
    2016-01-12 15:18:46 29954 [Note] Shutting down plugin 'XTRADB_INTERNAL_HASH_TABLES'
    2016-01-12 15:18:46 29954 [Note] Shutting down plugin 'XTRADB_READ_VIEW'
    2016-01-12 15:18:46 29954 [Note] Shutting down plugin 'InnoDB'
    2016-01-12 15:18:46 29954 [Note] InnoDB: FTS optimize thread exiting.
    2016-01-12 15:18:46 29954 [Note] InnoDB: Starting shutdown...
    2016-01-12 15:18:49 29954 [Note] InnoDB: Shutdown completed; log sequence number 94245881
    2016-01-12 15:18:49 29954 [Note] Shutting down plugin 'MyISAM'
    2016-01-12 15:18:49 29954 [Note] Shutting down plugin 'MRG_MYISAM'
    2016-01-12 15:18:49 29954 [Note] Shutting down plugin 'MEMORY'
    2016-01-12 15:18:49 29954 [Note] Shutting down plugin 'CSV'
    2016-01-12 15:18:49 29954 [Note] Shutting down plugin 'sha256_password'
    2016-01-12 15:18:49 29954 [Note] Shutting down plugin 'mysql_old_password'
    2016-01-12 15:18:49 29954 [Note] Shutting down plugin 'mysql_native_password'
    2016-01-12 15:18:49 29954 [Note] Shutting down plugin 'wsrep'
    2016-01-12 15:18:49 29954 [Note] Shutting down plugin 'binlog'
    2016-01-12 15:18:49 29954 [Note] /usr/sbin/mysqld: Shutdown complete

    160112 15:18:49 mysqld_safe WSREP: Recovered position 51c9a3db-aeaa-11e3-a5e6-bffb1faf3524:1320050
    Log of wsrep recovery (--wsrep-recover):
    2016-01-12 15:17:30 29954 [Note] Plugin 'FEDERATED' is disabled.
    InnoDB: Warning: innodb_log_block_size has been changed from default value 512. (###EXPERIMENTAL### operation)
    InnoDB: The log block size is set to 4096.
    2016-01-12 15:17:30 29954 [Note] InnoDB: Using atomics to ref count buffer pool pages
    2016-01-12 15:17:30 29954 [Note] InnoDB: The InnoDB memory heap is disabled
    2016-01-12 15:17:30 29954 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
    2016-01-12 15:17:30 29954 [Note] InnoDB: Memory barrier is not used
    2016-01-12 15:17:30 29954 [Note] InnoDB: Compressed tables use zlib 1.2.3.4
    2016-01-12 15:17:30 29954 [Note] InnoDB: Using Linux native AIO
    2016-01-12 15:17:30 29954 [Note] InnoDB: Using CPU crc32 instructions
    2016-01-12 15:17:30 29954 [Note] InnoDB: Initializing buffer pool, size = 10.0G
    2016-01-12 15:17:30 29954 [Note] InnoDB: Completed initialization of buffer pool
    2016-01-12 15:17:31 29954 [Note] InnoDB: Highest supported file format is Barracuda.
    2016-01-12 15:17:31 29954 [Note] InnoDB: The log sequence numbers 94245707 and 94245707 in ibdata files do not match the log sequence number 94245871 in the ib_logfiles!
    2016-01-12 15:17:31 29954 [Note] InnoDB: Database was not shutdown normally!
    2016-01-12 15:17:31 29954 [Note] InnoDB: Starting crash recovery.
    2016-01-12 15:17:31 29954 [Note] InnoDB: Reading tablespace information from the .ibd files...
    2016-01-12 15:18:46 29954 [Note] InnoDB: Restoring possible half-written data pages
    2016-01-12 15:18:46 29954 [Note] InnoDB: from the doublewrite buffer...
    2016-01-12 15:18:46 29954 [Note] InnoDB: 128 rollback segment(s) are active.
    2016-01-12 15:18:46 29954 [Note] InnoDB: Waiting for purge to start
    2016-01-12 15:18:46 29954 [Warning] InnoDB: Setting thread 30372 nice to -10 failed, current nice 0, errno 13
    2016-01-12 15:18:46 29954 [Warning] InnoDB: Setting thread 30373 nice to -10 failed, current nice 0, errno 13
    2016-01-12 15:18:46 29954 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.27-76.0 started; log sequence number 94245871
    2016-01-12 15:18:46 29954 [Warning] InnoDB: Skipping buffer pool dump/restore during wsrep recovery.
    2016-01-12 15:18:46 29954 [Note] RSA private key file not found: /srv/mysql/data//private_key.pem. Some authentication plugins will not work.
    2016-01-12 15:18:46 29954 [Note] RSA public key file not found: /srv/mysql/data//public_key.pem. Some authentication plugins will not work.
    2016-01-12 15:18:46 29954 [Note] Server hostname (bind-address): '192.168.10.12'; port: 3306
    2016-01-12 15:18:46 29954 [Note] - '192.168.10.12' resolves to '192.168.10.12';
    2016-01-12 15:18:46 29954 [Note] Server socket created on IP: '192.168.10.12'.
    2016-01-12 15:18:46 29954 [Note] WSREP: Recovered position: 51c9a3db-aeaa-11e3-a5e6-bffb1faf3524:1320050
    2016-01-12 15:18:46 29954 [Note] Binlog end
    2016-01-12 15:18:46 29954 [Note] Shutting down plugin 'partition'
    2016-01-12 15:18:46 29954 [Note] Shutting down plugin 'BLACKHOLE'
    2016-01-12 15:18:46 29954 [Note] Shutting down plugin 'ARCHIVE'
    2016-01-12 15:18:46 29954 [Note] Shutting down plugin 'PERFORMANCE_SCHEMA'
    2016-01-12 15:18:46 29954 [Note] Shutting down plugin 'INNODB_CHANGED_PAGES'
    2016-01-12 15:18:46 29954 [Note] Shutting down plugin 'INNODB_SYS_DATAFILES'
    2016-01-12 15:18:46 29954 [Note] Shutting down plugin 'INNODB_SYS_TABLESPACES'
    2016-01-12 15:18:46 29954 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN_COLS'
    2016-01-12 15:18:46 29954 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN'
    2016-01-12 15:18:46 29954 [Note] Shutting down plugin 'INNODB_SYS_FIELDS'
    2016-01-12 15:18:46 29954 [Note] Shutting down plugin 'INNODB_SYS_COLUMNS'
    2016-01-12 15:18:46 29954 [Note] Shutting down plugin 'INNODB_SYS_INDEXES'
    2016-01-12 15:18:46 29954 [Note] Shutting down plugin 'INNODB_SYS_TABLESTATS'
    2016-01-12 15:18:46 29954 [Note] Shutting down plugin 'INNODB_SYS_TABLES'
    2016-01-12 15:18:46 29954 [Note] Shutting down plugin 'INNODB_FT_INDEX_TABLE'
    2016-01-12 15:18:46 29954 [Note] Shutting down plugin 'INNODB_FT_INDEX_CACHE'
    2016-01-12 15:18:46 29954 [Note] Shutting down plugin 'INNODB_FT_CONFIG'
    2016-01-12 15:18:46 29954 [Note] Shutting down plugin 'INNODB_FT_BEING_DELETED'
    2016-01-12 15:18:46 29954 [Note] Shutting down plugin 'INNODB_FT_DELETED'
    2016-01-12 15:18:46 29954 [Note] Shutting down plugin 'INNODB_FT_DEFAULT_STOPWORD'
    2016-01-12 15:18:46 29954 [Note] Shutting down plugin 'INNODB_METRICS'
    2016-01-12 15:18:46 29954 [Note] Shutting down plugin 'INNODB_BUFFER_POOL_STATS'
    2016-01-12 15:18:46 29954 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE_LRU'
    2016-01-12 15:18:46 29954 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE'
    2016-01-12 15:18:46 29954 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX_RESET'
    2016-01-12 15:18:46 29954 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX'
    2016-01-12 15:18:46 29954 [Note] Shutting down plugin 'INNODB_CMPMEM_RESET'
    2016-01-12 15:18:46 29954 [Note] Shutting down plugin 'INNODB_CMPMEM'
    2016-01-12 15:18:46 29954 [Note] Shutting down plugin 'INNODB_CMP_RESET'
    2016-01-12 15:18:46 29954 [Note] Shutting down plugin 'INNODB_CMP'
    2016-01-12 15:18:46 29954 [Note] Shutting down plugin 'INNODB_LOCK_WAITS'
    2016-01-12 15:18:46 29954 [Note] Shutting down plugin 'INNODB_LOCKS'
    2016-01-12 15:18:46 29954 [Note] Shutting down plugin 'INNODB_TRX'
    2016-01-12 15:18:46 29954 [Note] Shutting down plugin 'XTRADB_RSEG'
    2016-01-12 15:18:46 29954 [Note] Shutting down plugin 'XTRADB_INTERNAL_HASH_TABLES'
    2016-01-12 15:18:46 29954 [Note] Shutting down plugin 'XTRADB_READ_VIEW'
    2016-01-12 15:18:46 29954 [Note] Shutting down plugin 'InnoDB'
    2016-01-12 15:18:46 29954 [Note] InnoDB: FTS optimize thread exiting.
    2016-01-12 15:18:46 29954 [Note] InnoDB: Starting shutdown...
    2016-01-12 15:18:49 29954 [Note] InnoDB: Shutdown completed; log sequence number 94245881
    2016-01-12 15:18:49 29954 [Note] Shutting down plugin 'MyISAM'
    2016-01-12 15:18:49 29954 [Note] Shutting down plugin 'MRG_MYISAM'
    2016-01-12 15:18:49 29954 [Note] Shutting down plugin 'MEMORY'
    2016-01-12 15:18:49 29954 [Note] Shutting down plugin 'CSV'
    2016-01-12 15:18:49 29954 [Note] Shutting down plugin 'sha256_password'
    2016-01-12 15:18:49 29954 [Note] Shutting down plugin 'mysql_old_password'
    2016-01-12 15:18:49 29954 [Note] Shutting down plugin 'mysql_native_password'
    2016-01-12 15:18:49 29954 [Note] Shutting down plugin 'wsrep'
    2016-01-12 15:18:49 29954 [Note] Shutting down plugin 'binlog'
    2016-01-12 15:18:49 29954 [Note] /usr/sbin/mysqld: Shutdown complete

    2016-01-12 15:18:50 0 [Note] /usr/sbin/mysqld (mysqld 5.6.27-76.0-56) starting as process 30388 ...
    2016-01-12 15:18:50 30388 [Note] WSREP: Read nil XID from storage engines, skipping position init
    2016-01-12 15:18:50 30388 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/libgalera_smm.so'
    2016-01-12 15:18:50 30388 [Note] WSREP: wsrep_load(): Galera 3.13(rb4bea65) by Codership Oy <[email protected]> loaded successfully.
    2016-01-12 15:18:50 30388 [Note] WSREP: CRC-32C: using hardware acceleration.
    2016-01-12 15:18:50 30388 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1
    2016-01-12 15:18:50 30388 [Note] WSREP: Passing config to GCS: base_dir = /srv/mysql/data/; base_host = 192.168.10.12; base_port = 4567; cert.log_conflicts = no; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /srv/mysql/data/; gcache.keep_pages_count = 0; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /srv/mysql/data//galera.cache; gcache.page_size = 128M; gcache.size = 128M; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc.checksum = false; pc.ignore_quor
    2016-01-12 15:18:51 30388 [Note] WSREP: Service thread queue flushed.
    2016-01-12 15:18:51 30388 [Note] WSREP: Assign initial position for certification: -1, protocol version: -1
    2016-01-12 15:18:51 30388 [Note] WSREP: wsrep_sst_grab()
    2016-01-12 15:18:51 30388 [Note] WSREP: Start replication
    2016-01-12 15:18:51 30388 [Note] WSREP: Setting initial position to 00000000-0000-0000-0000-000000000000:-1
    2016-01-12 15:18:51 30388 [Note] WSREP: protonet asio version 0
    2016-01-12 15:18:51 30388 [Note] WSREP: Using CRC-32C for message checksums.
    2016-01-12 15:18:51 30388 [Note] WSREP: backend: asio
    2016-01-12 15:18:51 30388 [Warning] WSREP: access file(/srv/mysql/data//gvwstate.dat) failed(No such file or directory)
    2016-01-12 15:18:51 30388 [Note] WSREP: restore pc from disk failed
    2016-01-12 15:18:51 30388 [Note] WSREP: GMCast version 0
    2016-01-12 15:18:51 30388 [Note] WSREP: (672f38ea, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567
    2016-01-12 15:18:51 30388 [Note] WSREP: (672f38ea, 'tcp://0.0.0.0:4567') multicast: , ttl: 1
    2016-01-12 15:18:51 30388 [Note] WSREP: EVS version 0
    2016-01-12 15:18:51 30388 [Note] WSREP: gcomm: connecting to group 'prestaserver_cluster', peer '192.168.10.11:,192.168.10.12:,192.168.10.13:'
    2016-01-12 15:18:51 30388 [Warning] WSREP: (672f38ea, 'tcp://0.0.0.0:4567') address 'tcp://192.168.10.12:4567' points to own listening address, blacklisting
    2016-01-12 15:18:51 30388 [Note] WSREP: (672f38ea, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers:
    2016-01-12 15:18:51 30388 [Note] WSREP: declaring 4eb463f5 at tcp://192.168.10.13:4567 stable
    2016-01-12 15:18:51 30388 [Note] WSREP: declaring e8566194 at tcp://192.168.10.11:4567 stable
    2016-01-12 15:18:51 30388 [Note] WSREP: Node 4eb463f5 state prim
    2016-01-12 15:18:51 30388 [Note] WSREP: view(view_id(PRIM,4eb463f5,35) memb {
    4eb463f5,0
    672f38ea,0
    e8566194,0
    } joined {
    } left {
    } partitioned {
    })
    2016-01-12 15:18:51 30388 [Note] WSREP: save pc into disk
    2016-01-12 15:18:52 30388 [Note] WSREP: gcomm: connected
    2016-01-12 15:18:52 30388 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636
    2016-01-12 15:18:52 30388 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)
    2016-01-12 15:18:52 30388 [Note] WSREP: Opened channel 'prestaserver_cluster'
    2016-01-12 15:18:52 30388 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 3
    2016-01-12 15:18:52 30388 [Note] WSREP: Waiting for SST to complete.
    2016-01-12 15:18:52 30388 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
    2016-01-12 15:18:52 30388 [Note] WSREP: STATE EXCHANGE: sent state msg: 677c4c97-b937-11e5-ac6f-3f661196e91d
    2016-01-12 15:18:52 30388 [Note] WSREP: STATE EXCHANGE: got state msg: 677c4c97-b937-11e5-ac6f-3f661196e91d from 0 (server3in)
    2016-01-12 15:18:52 30388 [Note] WSREP: STATE EXCHANGE: got state msg: 677c4c97-b937-11e5-ac6f-3f661196e91d from 2 (server1in)
    2016-01-12 15:18:52 30388 [Note] WSREP: STATE EXCHANGE: got state msg: 677c4c97-b937-11e5-ac6f-3f661196e91d from 1 (server2in)
    2016-01-12 15:18:52 30388 [Note] WSREP: Quorum results:
    version = 3,
    component = PRIMARY,
    conf_id = 34,
    members = 2/3 (joined/total),
    act_id = 1320125,
    last_appl. = -1,
    protocols = 0/7/3 (gcs/repl/appl),
    group UUID = 51c9a3db-aeaa-11e3-a5e6-bffb1faf3524
    2016-01-12 15:18:52 30388 [Note] WSREP: Flow-control interval: [28, 28]
    2016-01-12 15:18:52 30388 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 1320125)
    2016-01-12 15:18:52 30388 [Note] WSREP: State transfer required:
    Group state: 51c9a3db-aeaa-11e3-a5e6-bffb1faf3524:1320125
    Local state: 00000000-0000-0000-0000-000000000000:-1
    2016-01-12 15:18:52 30388 [Note] WSREP: New cluster view: global state: 51c9a3db-aeaa-11e3-a5e6-bffb1faf3524:1320125, view# 35: Primary, number of nodes: 3, my index: 1, protocol version 3
    2016-01-12 15:18:52 30388 [Warning] WSREP: Gap in state sequence. Need state transfer.
    2016-01-12 15:18:52 30388 [Note] WSREP: Running: 'wsrep_sst_xtrabackup-v2 --role 'joiner' --address '192.168.10.12' --datadir '/srv/mysql/data/' --defaults-file '/etc/mysql/my.cnf' --defaults-group-suffix '' --parent '30388' '' '
    WSREP_SST: [INFO] Streaming with xbstream (20160112 15:18:52.134)
    WSREP_SST: [INFO] Using socat as streamer (20160112 15:18:52.135)
    WSREP_SST: [INFO] Stale sst_in_progress file: /srv/mysql/data//sst_in_progress (20160112 15:18:52.139)
    WSREP_SST: [INFO] Evaluating timeout -k 110 100 socat -u TCP-LISTEN:4444,reuseaddr stdio | xbstream -x; RC=( ${PIPESTATUS[@]} ) (20160112 15:18:52.158)
    2016-01-12 15:18:52 30388 [Note] WSREP: Prepared SST request: xtrabackup-v2|192.168.10.12:4444/xtrabackup_sst//1
    2016-01-12 15:18:52 30388 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
    2016-01-12 15:18:52 30388 [Note] WSREP: REPL Protocols: 7 (3, 2)
    2016-01-12 15:18:52 30388 [Note] WSREP: Service thread queue flushed.
    2016-01-12 15:18:52 30388 [Note] WSREP: Assign initial position for certification: 1320125, protocol version: 3
    2016-01-12 15:18:52 30388 [Note] WSREP: Service thread queue flushed.
    2016-01-12 15:18:52 30388 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (51c9a3db-aeaa-11e3-a5e6-bffb1faf3524): 1 (Operation not permitted)
    at galera/src/replicator_str.cpp:prepare_for_IST():489. IST will be unavailable.
    2016-01-12 15:18:52 30388 [Note] WSREP: Member 1.0 (server2in) requested state transfer from '*any*'. Selected 0.0 (server3in)(SYNCED) as donor.
    2016-01-12 15:18:52 30388 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 1320125)
    2016-01-12 15:18:52 30388 [Note] WSREP: Requesting state transfer: success, donor: 0
    WSREP_SST: [ERROR] Parent mysqld process (PID:30388) terminated unexpectedly. (20160112 15:18:52.565)
    WSREP_SST: [ERROR] Cleanup after exit with status:32 (20160112 15:18:52.567)
    2016-01-12 15:18:52 30388 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'joiner' --address '192.168.10.12' --datadir '/srv/mysql/data/' --defaults-file '/etc/mysql/my.cnf' --defaults-group-suffix '' --parent '30388' '' : 32 (Broken pipe)
    2016-01-12 15:18:52 30388 [ERROR] WSREP: Failed to read uuid:seqno from joiner script.
    2016-01-12 15:18:52 30388 [ERROR] WSREP: SST script aborted with error 32 (Broken pipe)
    2016-01-12 15:18:52 30388 [ERROR] WSREP: SST failed: 32 (Broken pipe)
    2016-01-12 15:18:52 30388 [ERROR] Aborting

    2016-01-12 15:18:54 30388 [Note] WSREP: (672f38ea, 'tcp://0.0.0.0:4567') turning message relay requesting off
    2016-01-12 15:18:54 30388 [Note] WSREP: Closing send monitor...
    2016-01-12 15:18:54 30388 [Note] WSREP: Closed send monitor.
    2016-01-12 15:18:54 30388 [Note] WSREP: gcomm: terminating thread
    2016-01-12 15:18:54 30388 [Note] WSREP: gcomm: joining thread
    2016-01-12 15:18:54 30388 [Note] WSREP: gcomm: closing backend
    2016-01-12 15:18:54 30388 [Note] WSREP: view(view_id(NON_PRIM,4eb463f5,35) memb {
    672f38ea,0
    } joined {
    } left {
    } partitioned {
    4eb463f5,0
    e8566194,0
    })
    2016-01-12 15:18:54 30388 [Note] WSREP: view((empty))
    2016-01-12 15:18:54 30388 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
    2016-01-12 15:18:54 30388 [Note] WSREP: gcomm: closed
    2016-01-12 15:18:54 30388 [Note] WSREP: Flow-control interval: [16, 16]
    2016-01-12 15:18:54 30388 [Note] WSREP: Received NON-PRIMARY.
    2016-01-12 15:18:54 30388 [Note] WSREP: Shifting JOINER -> OPEN (TO: 1320125)
    2016-01-12 15:18:54 30388 [Note] WSREP: Received self-leave message.
    2016-01-12 15:18:54 30388 [Note] WSREP: Flow-control interval: [0, 0]
    2016-01-12 15:18:54 30388 [Note] WSREP: Received SELF-LEAVE. Closing connection.
    2016-01-12 15:18:54 30388 [Note] WSREP: Shifting OPEN -> CLOSED (TO: 1320125)
    2016-01-12 15:18:54 30388 [Note] WSREP: RECV thread exiting 0: Success
    2016-01-12 15:18:54 30388 [Note] WSREP: recv_thread() joined.
    2016-01-12 15:18:54 30388 [Note] WSREP: Closing replication queue.
    2016-01-12 15:18:54 30388 [Note] WSREP: Closing slave action queue.
    2016-01-12 15:18:54 30388 [Note] WSREP: Service disconnected.
    2016-01-12 15:18:54 30388 [Note] WSREP: rollbacker thread exiting
    2016-01-12 15:18:55 30388 [Note] WSREP: Some threads may fail to exit.
    2016-01-12 15:18:55 30388 [Note] Binlog end
    2016-01-12 15:18:55 30388 [Note] /usr/sbin/mysqld: Shutdown complete

    Error in my_thread_global_end(): 1 threads didn't exit
    160112 15:19:00 mysqld_safe mysqld from pid file /srv/mysql/data/server2.pid ended

    Thanks in advance
  • PercUsePercUse Contributor Inactive User Role Beginner
    Hi Guys,

    I have the feeling that there's not anymore someone who can answer our questions and it's the case for different topic here not only mine...

    Regards,
Sign In or Register to comment.

MySQL, InnoDB, MariaDB and MongoDB are trademarks of their respective owners.
Copyright ©2005 - 2020 Percona LLC. All rights reserved.