# Slave replication failure after full SST on master **URL:** \ **Category:** Percona XtraDB Cluster 5.x\ **Created:** [September 15, 2023, 5:58am UTC](https://forums.percona.com/t/slave-replication-failure-after-full-sst-on-master/25197 "2023-09-15T05:58:46Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![vrcsts](https://avatars.discourse-cdn.com/v4/letter/v/58f4c7/32.png) [@vrcsts](https://forums.percona.com/u/vrcsts)\ **Post date:** [September 15, 2023, 5:58am UTC](https://forums.percona.com/t/slave-replication-failure-after-full-sst-on-master/25197/1 "2023-09-15T05:58:46Z")
Hello, I have a set up consisting of a 3 node PXC cluster with 5.7.33, let’s call the nodes my01, my02, my03. The cluster itself is single master, using ProxySQL to direct writes to one single server. Then, my03 is replicating to a fourth single MySQL in a different location using MySQL GTID based replication. Let’s call this fourth server myreplica01. This setup works perfectly until an SST is done to my03 for some reason. At this point, the replication stops working and I have to configure it again from scratch (full backup on my03, move the backup to myreplica01, restore, stat replication). Slave status shows something like this: ```auto Coordinator stopped because there were error(s) in the worker(s). The most recent failure being: Worker 1 failed executing transaction 'f2d83a3e-570b-ee18-5d2e-596888ce4869:1016331531' at master log mysql-bin.008521, end_log_pos 197880. See error log and/or performance_schema.replication_applier_status_by_worker table for more details about this failure or others, if any. ``` This behavior makes _some_ sense, as the bin-logs on my03 are reset by the SST, but given that the database itself is _the same_, is there a way to keep master-slave replication working even after SST? Is there a faster way to restore replication without setting it up from scratch and without losing any transaction/data? Thanks a lot ---
**Author:** ![Ivan\_Groenewold](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/ivan_groenewold/32/6299_2.png) [@Ivan\_Groenewold](https://forums.percona.com/u/Ivan_Groenewold)\ **Post date:** [September 15, 2023, 12:52pm UTC](https://forums.percona.com/t/slave-replication-failure-after-full-sst-on-master/25197/2 "2023-09-15T12:52:40Z")
Hi, the way to accomplish this would be enabling GTID. This way if the replication source has an issue, you can easily repoint the replica to a different source. See [How to setup a PXC cluster with GTIDs](https://www.percona.com/blog/how-to-setup-a-pxc-cluster-with-gtids-and-have-async-slaves-replicating-from-it/) for details Beginning with MySQL 8.0.22 you can also configure [automatic connection failover](https://dev.mysql.com/doc/refman/8.0/en/replication-asynchronous-connection-failover.html), so no manual repointing will be needed anymore. ---
**Author:** ![matthewb](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/matthewb/32/34_2.png) [@matthewb](https://forums.percona.com/u/matthewb)\ **Post date:** [September 15, 2023, 3:30pm UTC](https://forums.percona.com/t/slave-replication-failure-after-full-sst-on-master/25197/3 "2023-09-15T15:30:37Z")
Configure server\_uuid for each PXC node, instead of letting MySQL pick one at random each time an SST happens. This should keep the GTID set the same, even after restore. The better question to be asking is “why is my03 executing an SST?” That should not be happening under normal operations. If my03 goes down and back up, it should **IST** first. If your IST isn’t working, check your galera gcache size. ---
**Author:** ![vrcsts](https://avatars.discourse-cdn.com/v4/letter/v/58f4c7/32.png) [@vrcsts](https://forums.percona.com/u/vrcsts)\ **Post date:** [September 16, 2023, 10:18am UTC](https://forums.percona.com/t/slave-replication-failure-after-full-sst-on-master/25197/4 "2023-09-16T10:18:56Z")
The value of server\_uuid is placed in $datadir/auto.cnf and that file is removed by SST, so MySQL creates a new server\_uuid when it starts after the SST. Don’t know how to set a fixed server\_uuid. This time the SST happened because I was messing with compressor/decompressor options and forgot to install zstd on this node ☹ IST is used everytime I had to reboot this node, so state transfer is working fine. ---
**Author:** ![matthewb](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/matthewb/32/34_2.png) [@matthewb](https://forums.percona.com/u/matthewb)\ **Post date:** [September 17, 2023, 1:58pm UTC](https://forums.percona.com/t/slave-replication-failure-after-full-sst-on-master/25197/5 "2023-09-17T13:58:07Z")
You can set `server_uuid` in the my.cnf ---
**Author:** ![vrcsts](https://avatars.discourse-cdn.com/v4/letter/v/58f4c7/32.png) [@vrcsts](https://forums.percona.com/u/vrcsts)\ **Post date:** [September 21, 2023, 11:21am UTC](https://forums.percona.com/t/slave-replication-failure-after-full-sst-on-master/25197/6 "2023-09-21T11:21:46Z")
Seems like no, I can’t set it. At least on Ubuntu 22.04 with Percona MySQL 5.7.43. I may be doing something wrong, but: - If set on `/etc/mysql/percona-xtradb-cluster.conf.d/mysqld.cnf` as: ```auto server_uuid=some-uuid ``` Mysql does not start and complains about unknown variable server\_uuid. - When set on `/etc/mysql/percona-xtradb-cluster.conf.d/mysqld.cnf` as: ```auto [auto] server_uuid=some-uuid ``` Mysql starts but the uuid shown by `select @@server_uuid` shows the uuid in $datadir/auto.cnf instead of that in mysql.cnf. - If I remove `$datadir/auto.cnf` and restart Mysql, a new auto.cnf file is created with a different uuid. The [documentation](https://dev.mysql.com/doc/refman/5.7/en/replication-options.html#sysvar_server_uuid) does not mention that server\_uuid can be read nor set in any other file or way but in auto.cnf. Given that such file is deleted during an SST, I see no way to keep the same server\_uuid in order to avoid rebuilding the replication from scratch if an SST happens on master. ---
**Author:** ![matthewb](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/matthewb/32/34_2.png) [@matthewb](https://forums.percona.com/u/matthewb)\ **Post date:** [September 21, 2023, 4:02pm UTC](https://forums.percona.com/t/slave-replication-failure-after-full-sst-on-master/25197/7 "2023-09-21T16:02:04Z")
You are correct in that server\_uuid doesn’t work inside my.cnf, but it does work inside auto.cnf in my testing of 5.7.38. The SELECT @@server\_uuid matched what was in auto.cnf. In any case, since this file is replaced by SST, we are back to original issue. If mysql04 is using AUTO\_POSITION=1, when replication breaks from 03-\>04, you should be able to go to 04 and simply CHANGE MASTER to mysql02 and resume. Because GTIDs are in use everywhere, the same GTID set should exist on mysql02 and replication should resume.