Not the answer you need?
Register and ask your own question!

First node fail to start after reboot when cluster under havy load

mariuszmmariuszm EntrantCurrent User Role Beginner
Hi! I did setup simple 3 node cluster. When there is no load everything works like a harm. Let's say nodes name are: A B C, and this is order of start. A with bootstrap-pxe command.
When I load mysql dump(2GB file to generate havy load) file on let's say node B and then do gracefull stop and start on node A it fails to start. Problem is that I have to stop all nodes and then I can start node A and B and so on.
Does is work like design or it is some kind of bug? Does anyone know how to recover from this situation without reboot whole cluster?
This is what I see in logs:
root@nodeA:~# cat /var/lib/mysql/grastate.dat
# GALERA saved state
version: 2.1
uuid:    00000000-0000-0000-0000-000000000000
seqno:   -1

2016-03-18 04:28:09 5076 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (25cdfc4c-ecdd-11e5-a76c-3e8517058066): 1 (Operation not
         at galera/src/replicator_str.cpp:prepare_for_IST():489. IST will be unavailable.
2016-03-18 04:28:09 5076 [Note] WSREP: Member 1.0 () requested state transfer from '*any*'. Selected 0.0 ()(SYNCED) as donor.
2016-03-18 04:28:09 5076 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 551)
2016-03-18 04:28:09 5076 [Note] WSREP: Requesting state transfer: success, donor: 0
2016-03-18 04:28:09 5076 [Warning] WSREP: 0.0 (): State transfer to 1.0 () failed: -12 (Cannot allocate memory)
2016-03-18 04:28:09 5076 [ERROR] WSREP: gcs/src/gcs_group.cpp:gcs_group_handle_join_msg():733: Will never receive state. Need to abort.

dpkg -l | grep percona
ii  percona-xtrabackup                2.3.3-1.jessie              amd64        Open source backup tool for InnoDB and XtraDB
ii  percona-xtradb-cluster-56         5.6.28-25.14-1.jessie       amd64        Percona XtraDB Cluster with Galera
ii  percona-xtradb-cluster-client-5.6 5.6.28-25.14-1.jessie       amd64        Percona XtraDB Cluster database client binaries
ii  percona-xtradb-cluster-common-5.6 5.6.28-25.14-1.jessie       amd64        Percona XtraDB Cluster database common files (e.g. /etc/mysql/my.cnf)
ii  percona-xtradb-cluster-galera-3   3.14-1.jessie               amd64        Metapackage for latest version of galera3.
ii  percona-xtradb-cluster-galera-3.x 3.14-1.jessie               amd64        Galera components of Percona XtraDB Cluster
ii  percona-xtradb-cluster-server-5.6 5.6.28-25.14-1.jessie       amd64        Percona XtraDB Cluster database server binaries

Also I see this:
2016-03-18 05:25:17 1017 [Note] WSREP: Running: 'wsrep_sst_rsync --role 'donor' --address '' --socket '/var/run/mysqld/mysqld.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/mysql/my.cnf' --defaults-group-suffix ''   '' --gtid '25cdfc4c-ecdd-11e5-a76c-3e8517058066:2404''
2016-03-18 05:25:17 1017 [ERROR] WSREP: fork() failed: 12 (Cannot allocate memory)

Sign In or Register to comment.

MySQL, InnoDB, MariaDB and MongoDB are trademarks of their respective owners.
Copyright ©2005 - 2020 Percona LLC. All rights reserved.