We have a basic master/slave setup. One master and three slaves. Replication was working flawless for months.
They were all on version 5.6.14-56-log Percona Server (GPL).
We added a new slave on a VM a few days ago. The server was build new and the latest version of MySQL was installed 5.6.21-70.0-log Percona Server (GPL).
To set up the slave I used xtrabackup off an existing slave. But before I did I updated the two slaves to 5.6.21-70.0-log Percona Server (GPL) so that they were all on the
same version and the master was still on 5.6.14. We didn’t want to run any updates on the live production server.
All servers were replicating fine for three days until today when suddenly all three slaves stopped with no errors. IO and SQL thread all died but there was no replication error. Issuing START SLAVE did nothing and did not generate a last error. The master shows no slave threads, there are no errors in the log, and its running fine.
Looking through the mysqld.log all the servers had this when issuing START SLAVE:
2014-11-14 20:50:28 15361 [Note] Slave SQL thread initialized, starting replication in log ‘mysql-bin.000244’ at position 775127000, relay log ‘/var/lib/mysql/relay-bin.000007’ position: 775127163
04:50:29 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Please help us make Percona Server better by reporting any
bugs at [ http://bugs.percona.com/ ]
Now I don’t know what’s wrong. Signal 11 seems pretty ambiguous in searches.
Did replication finally hit a snag over the difference in versions? Is there a problem with the master?
Is it safe to just run an update on the master and try to start the slaves again? Does replication need to be reset on all the servers?
Thanks for any help.