# Can't reconnect to cluster after reboot
**URL:** https://forums.percona.com/t/cant-reconnect-to-cluster-after-reboot/28141
**Category:** Percona XtraDB Cluster 5.x
**Created:** [January 30, 2024, 8:17pm UTC](https://forums.percona.com/t/cant-reconnect-to-cluster-after-reboot/28141 "2024-01-30T20:17:37Z")
**Posts on this page:** 16
**Page:** 1
### Author:  [@Ken\_Scott](https://forums.percona.com/u/Ken_Scott)
#### Post date: [January 30, 2024, 8:17pm UTC](https://forums.percona.com/t/cant-reconnect-to-cluster-after-reboot/28141/1 "2024-01-30T20:17:37Z")
This is now happening on 2 of my 5 nodes. We updating a mailer program on reboot the node can’t join back to the cluster. The only way i know this solve this is to blow away the machine and start from scratch.
Joiner node:
root@webnode2:/home/ken# /etc/init.d/mysql start
Starting mysql (via systemctl): mysql.serviceJob for mysql.service failed because the control process exited with error code.
See “systemctl status mysql.service” and “journalctl -xe” for details.
failed!
root@webnode2:/home/ken# systemctl status mysql.service
● mysql.service - LSB: Start and stop the mysql (Percona XtraDB Cluster) daemon
Loaded: loaded (/etc/init.d/mysql; generated)
Active: failed (Result: exit-code) since Tue 2024-01-30 15:12:55 EST; 40s ago
Docs: man:systemd-sysv-generator(8)
Process: 3979495 ExecStart=/etc/init.d/mysql start (code=exited, status=1/FAILURE)
Jan 30 15:12:35 [webnode2.long-mcquade.com](http://webnode2.long-mcquade.com) systemd[1]: Starting LSB: Start and stop the mysql (Percona XtraDB Cluster) daemon…
Jan 30 15:12:35 [webnode2.long-mcquade.com](http://webnode2.long-mcquade.com) mysql[3979495]: \* Stale sst\_in\_progress file in datadir mysqld
Jan 30 15:12:35 [webnode2.long-mcquade.com](http://webnode2.long-mcquade.com) mysql[3979495]: \* Starting MySQL (Percona XtraDB Cluster) database server mysqld
Jan 30 15:12:35 [webnode2.long-mcquade.com](http://webnode2.long-mcquade.com) mysql[3979495]: \* State transfer in progress, setting sleep higher mysqld
Jan 30 15:12:55 [webnode2.long-mcquade.com](http://webnode2.long-mcquade.com) mysql[3979495]: \* The server quit without updating PID file (/var/run/mysqld/mysqld.pid).
Jan 30 15:12:55 [webnode2.long-mcquade.com](http://webnode2.long-mcquade.com) mysql[3979495]: …fail!
Jan 30 15:12:55 [webnode2.long-mcquade.com](http://webnode2.long-mcquade.com) systemd[1]: mysql.service: Control process exited, code=exited, status=1/FAILURE
Jan 30 15:12:55 [webnode2.long-mcquade.com](http://webnode2.long-mcquade.com) systemd[1]: mysql.service: Failed with result ‘exit-code’.
Jan 30 15:12:55 [webnode2.long-mcquade.com](http://webnode2.long-mcquade.com) systemd[1]: Failed to start LSB: Start and stop the mysql (Percona XtraDB Cluster) daemon.
Doner Node:
2024-01-30T20:12:41.941906Z 0 [Note] WSREP: Member 2.0 (pxc2) requested state transfer from ‘_any_’. Selected 0.0 (pxc3)(SYNCED) as donor.
2024-01-30T20:12:43.332884Z 0 [Note] WSREP: (f23c4ba4, ‘tcp://0.0.0.0:4567’) turning message relay requesting off
2024-01-30T20:12:53.523896Z 0 [Warning] WSREP: 0.0 (pxc3): State transfer to 2.0 (pxc2) failed: -22 (Invalid argument)
2024-01-30T20:12:53.524856Z 0 [Note] WSREP: Member 0.0 (pxc3) synced with group.
2024-01-30T20:12:53.524909Z 0 [Note] WSREP: declaring 225a4946 at tcp://172.26.0.11:4567 stable
2024-01-30T20:12:53.524950Z 0 [Note] WSREP: declaring 38921ada at tcp://172.26.0.9:4567 stable
2024-01-30T20:12:53.524989Z 0 [Note] WSREP: forgetting eb189faa (tcp://172.26.0.12:4567)
2024-01-30T20:12:53.525615Z 0 [Note] WSREP: Node 225a4946 state primary
2024-01-30T20:12:53.526106Z 0 [Note] WSREP: Current view of cluster as seen by this node
view (view\_id(PRIM,225a4946,183)
memb {
225a4946,0
38921ada,0
f23c4ba4,0
}
joined {
}
left {
}
partitioned {
eb189faa,0
}
)
2024-01-30T20:12:53.526129Z 0 [Note] WSREP: Save the discovered primary-component to disk
2024-01-30T20:12:53.526690Z 0 [Note] WSREP: forgetting eb189faa (tcp://172.26.0.12:4567)
2024-01-30T20:12:53.526773Z 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my\_idx = 2, memb\_num = 3
2024-01-30T20:12:53.526812Z 0 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
2024-01-30T20:12:53.527188Z 0 [Note] WSREP: STATE EXCHANGE: sent state msg: f32dbad8-bfab-11ee-b9bd-bbf4466ecd50
2024-01-30T20:12:53.527437Z 0 [Note] WSREP: STATE EXCHANGE: got state msg: f32dbad8-bfab-11ee-b9bd-bbf4466ecd50 from 0 (pxc3)
2024-01-30T20:12:53.527453Z 0 [Note] WSREP: STATE EXCHANGE: got state msg: f32dbad8-bfab-11ee-b9bd-bbf4466ecd50 from 1 (pxc5)
2024-01-30T20:12:53.527466Z 0 [Note] WSREP: STATE EXCHANGE: got state msg: f32dbad8-bfab-11ee-b9bd-bbf4466ecd50 from 2 (pxc6)
2024-01-30T20:12:53.527483Z 0 [Note] WSREP: Quorum results:
version = 6,
component = PRIMARY,
conf\_id = 171,
members = 3/3 (primary/total),
act\_id = 7130896577,
last\_appl. = 7130896493,
protocols = 0/9/3 (gcs/repl/appl),
group UUID = 3a0118b7-6c7b-11eb-935d-5fcc5d08be87
2024-01-30T20:12:53.527498Z 0 [Note] WSREP: Flow-control interval: [173, 173]
2024-01-30T20:12:53.527687Z 6 [Note] WSREP: REPL Protocols: 9 (4, 2)
2024-01-30T20:12:53.527725Z 6 [Note] WSREP: REPL Protocols: 9 (4, 2)
2024-01-30T20:12:53.527745Z 6 [Note] WSREP: New cluster view: global state: 3a0118b7-6c7b-11eb-935d-5fcc5d08be87:7130896577, view# 172: Primary, number of nodes: 3, my index: 2, protocol version 3
2024-01-30T20:12:53.527756Z 6 [Note] WSREP: Setting wsrep\_ready to true
2024-01-30T20:12:53.527767Z 6 [Note] WSREP: Auto Increment Offset/Increment re-align with cluster membership change (Offset: 4 → 3) (Increment: 4 → 3)
2024-01-30T20:12:53.528755Z 6 [Note] WSREP: wsrep\_notify\_cmd is not defined, skipping notification.
2024-01-30T20:12:53.531022Z 6 [Note] WSREP: Assign initial position for certification: 7130896577, protocol version: 4
2024-01-30T20:12:53.531133Z 0 [Note] WSREP: Service thread queue flushed.
2024-01-30T20:12:58.835552Z 0 [Note] WSREP: cleaning up eb189faa (tcp://172.26.0.12:4567)
Any thoughts on what I can do to resolve this?
---
### Author:  [@matthewb](https://forums.percona.com/u/matthewb)
#### Post date: [January 30, 2024, 9:02pm UTC](https://forums.percona.com/t/cant-reconnect-to-cluster-after-reboot/28141/2 "2024-01-30T21:02:47Z")
> [@Ken\_Scott](#):
>
> /etc/init.d/mysql start
This is extremely old way of managing services in linux. You should learn the new systemctl commands.
`systemctl start|stop|restart mysql`
`systemctl start|stop mysql@bootstrap`
> [@Ken\_Scott](#):
>
> - Stale sst\_in\_progress file in datadir mysqld
You need to erase the data directory and clean out whatever failed before and then start it again.
---
### Author:  [@Ken\_Scott](https://forums.percona.com/u/Ken_Scott)
#### Post date: [January 31, 2024, 7:33pm UTC](https://forums.percona.com/t/cant-reconnect-to-cluster-after-reboot/28141/3 "2024-01-31T19:33:35Z")
Hello, I used “systemctl start mysql” with the same result.
Also where can I find the data directory on Ubuntu?
I was running:
> sudo rm -rf /var/lib/mysql\* /var/lib/mysql-files\*
---
### Author:  [@matthewb](https://forums.percona.com/u/matthewb)
#### Post date: [January 31, 2024, 8:06pm UTC](https://forums.percona.com/t/cant-reconnect-to-cluster-after-reboot/28141/4 "2024-01-31T20:06:25Z")
/var/lib/mysql is the default directory. Stop mysql. Verify stopped using `ps -Af | grep mysqld`. Erase the datadir then start it back up. Please provide a copy of your my.cnf file.
---
### Author:  [@Ken\_Scott](https://forums.percona.com/u/Ken_Scott)
#### Post date: [January 31, 2024, 8:12pm UTC](https://forums.percona.com/t/cant-reconnect-to-cluster-after-reboot/28141/5 "2024-01-31T20:12:39Z")
I verified that MySQL was stopped and removed the data directory but still the same issue.
here is my config;
> !includedir /etc/mysql/conf.d/
> !includedir /etc/mysql/percona-xtradb-cluster.conf.d/
>
> [mysqld]
>
> interactive\_timeout = 10
> wait\_timeout = 10
>
> max\_connections = 505 # Values \< 1000 are typically good
> max\_user\_connections = 500 # Limit one specific user/application
> thread\_cache\_size = 505 # Up to max\_connections makes sense
>
> query\_cache\_type = 1
> query\_cache\_size = 100M
>
> innodb\_buffer\_pool\_size = 5G
>
> # Session variables
>
> sort\_buffer\_size = 10M # Could be too big for many small sorts
> tmp\_table\_size = 32M # Make sure your temporary results do NOT contain BLOB/TEXT attributes
>
> read\_buffer\_size = 20M # Resist to change this parameter if you do not know what you are doing
> read\_rnd\_buffer\_size = 10M # Resist to change this parameter if you do not know what you are doing
> join\_buffer\_size = 50M # Resist to change this parameter if you do not know what you are doing
>
> # Other buffers and caches
>
> table\_definition\_cache = 1400 # As big as many tables you have
> table\_open\_cache = 5000 # connections x tables/connection (~2)
> table\_open\_cache\_instances = 16 # New default in 5.7
>
> wsrep\_provider=/usr/lib/libgalera\_smm.so
>
> wsrep\_cluster\_name=pxc-cluster
> wsrep\_cluster\_address=gcomm://172.26.0.11,172.26.0.14,172.26.0.9
>
> wsrep\_node\_name=pxc2
> wsrep\_node\_address=172.26.0.12
>
> wsrep\_sst\_method=xtrabackup-v2
> wsrep\_sst\_auth=sstuser:passw0rd
> pxc\_strict\_mode=ENFORCING
> binlog\_format=ROW
> default\_storage\_engine=InnoDB
> innodb\_autoinc\_lock\_mode=3
---
### Author:  [@matthewb](https://forums.percona.com/u/matthewb)
#### Post date: [January 31, 2024, 9:05pm UTC](https://forums.percona.com/t/cant-reconnect-to-cluster-after-reboot/28141/6 "2024-01-31T21:05:52Z")
Ok. Just to confirm, you have 2 other nodes that are online and in PRIMARY state, correct? Then, please do the following:
1. stop mysql on failed node
2. erase datadir
3. erase mysql error log file
4. start mysql
after it fails, provide error log contents please.
---
### Author:  [@Ken\_Scott](https://forums.percona.com/u/Ken_Scott)
#### Post date: [January 31, 2024, 10:03pm UTC](https://forums.percona.com/t/cant-reconnect-to-cluster-after-reboot/28141/7 "2024-01-31T22:03:08Z")
Ok, I’ve done that;
> root@webnode2:/var/log# systemctl status mysql.service
> ● mysql.service - LSB: Start and stop the mysql (Percona XtraDB Cluster) daemon
> Loaded: loaded (/etc/init.d/mysql; generated)
> Active: failed (Result: exit-code) since Wed 2024-01-31 17:01:25 EST; 5s ago
> Docs: man:systemd-sysv-generator(8)
> Process: 594265 ExecStart=/etc/init.d/mysql start (code=exited, status=1/FAILURE)
>
> Jan 31 17:01:23 webnode2.test-.com systemd[1]: Starting LSB: Start and stop the mysql (Percona XtraDB Cluster) daemon…
> Jan 31 17:01:23 [webnode2.test.com](http://webnode2.test.com) mysql[594265]: \* Starting MySQL (Percona XtraDB Cluster) database server mysqld
> Jan 31 17:01:25 [webnode2.test.com](http://webnode2.test.com) mysql[594265]: \* The server quit without updating PID file (/var/run/mysqld/mysqld.pid).
> Jan 31 17:01:25 [webnode2.test.com](http://webnode2.test.com) mysql[594265]: …fail!
> Jan 31 17:01:25 [webnode2.test.com](http://webnode2.test.com) systemd[1]: mysql.service: Control process exited, code=exited, status=1/FAILURE
> Jan 31 17:01:25 [webnode2.test.com](http://webnode2.test.com) systemd[1]: mysql.service: Failed with result ‘exit-code’.
> Jan 31 17:01:25 [webnode2.test.com](http://webnode2.test.com) systemd[1]: Failed to start LSB: Start and stop the mysql (Percona XtraDB Cluster) daemon.
---
### Author:  [@matthewb](https://forums.percona.com/u/matthewb)
#### Post date: [January 31, 2024, 10:07pm UTC](https://forums.percona.com/t/cant-reconnect-to-cluster-after-reboot/28141/8 "2024-01-31T22:07:08Z")
Hello @Ken_Scott,
that is not the mysql error log. that is systemctl’s messages which never (usually) contain any helpful information. You need to find mysql’s actual error log and zero it out (step 3 above) and then copy/paste it here.
---
### Author:  [@Ken\_Scott](https://forums.percona.com/u/Ken_Scott)
#### Post date: [February 1, 2024, 8:46pm UTC](https://forums.percona.com/t/cant-reconnect-to-cluster-after-reboot/28141/9 "2024-02-01T20:46:46Z")
This is the MySQL error log:
> 2024-02-01T20:45:31.866610Z 2 [Note] WSREP: Prepared SST/IST request: xtrabackup-v2|172.26.0.12:4444/xtrabackup\_sst//1
> 2024-02-01T20:45:31.866680Z 2 [Note] WSREP: Auto Increment Offset/Increment re-align with cluster membership change (Offset: 1 → 3) (Increment: 1 → 4)
> 2024-02-01T20:45:31.866796Z 2 [Note] WSREP: wsrep\_notify\_cmd is not defined, skipping notification.
> 2024-02-01T20:45:31.866868Z 2 [Note] WSREP: Assign initial position for certification: 7149092259, protocol version: 4
> 2024-02-01T20:45:31.867012Z 0 [Note] WSREP: Service thread queue flushed.
> 2024-02-01T20:45:31.867109Z 2 [Note] WSREP: Check if state gap can be serviced using IST
> 2024-02-01T20:45:31.867158Z 2 [Note] WSREP: Local UUID: b64bc893-c142-11ee-a085-a68742f583c6 != Group UUID: 3a0118b7-6c7b-11eb-935d-5fcc5d08be87
> 2024-02-01T20:45:31.867277Z 2 [Note] WSREP: State gap can’t be serviced using IST. Switching to SST
> 2024-02-01T20:45:31.867316Z 2 [Note] WSREP: Failed to prepare for incremental state transfer: Local state UUID (b64bc893-c142-11ee-a085-a68742f583c6) does not match group state UUID (3a0118b7-6c7b-11eb-935d-5fcc5d08be87): 1 (Operation not permitted)
> at galera/src/replicator\_str.cpp:prepare\_for\_IST():518. IST will be unavailable.
> 2024-02-01T20:45:31.869147Z 0 [Note] WSREP: Member 2.0 (pxc2) requested state transfer from ‘_any_’. Selected 0.0 (pxc3)(SYNCED) as donor.
> 2024-02-01T20:45:31.869190Z 0 [Note] WSREP: Shifting PRIMARY → JOINER (TO: 7149092265)
> 2024-02-01T20:45:31.869249Z 2 [Note] WSREP: Requesting state transfer: success, donor: 0
> 2024-02-01T20:45:31.869305Z 2 [Note] WSREP: GCache history reset: b64bc893-c142-11ee-a085-a68742f583c6:0 → 3a0118b7-6c7b-11eb-935d-5fcc5d08be87:7149092259
> 2024-02-01T20:45:32.498536Z WSREP\_SST: [INFO] Streaming with xbstream
> 2024-02-01T20:45:32.510815Z WSREP\_SST: [INFO] Proceeding with SST…
> 2024-02-01T20:45:32.559746Z WSREP\_SST: [INFO] …Waiting for SST streaming to complete!
> 2024-02-01T20:45:33.417683Z 0 [Note] WSREP: (d61ae3cb, ‘tcp://0.0.0.0:4567’) turning message relay requesting off
> 2024-02-01T20:45:43.401239Z WSREP\_SST: [ERROR] \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\* FATAL ERROR \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*
> 2024-02-01T20:45:43.403995Z WSREP\_SST: [ERROR] xtrabackup\_checkpoints missing. xtrabackup/SST failed on DONOR. Check DONOR log
> 2024-02-01T20:45:43.406751Z WSREP\_SST: [ERROR] \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*
> 2024-02-01T20:45:43.409748Z WSREP\_SST: [ERROR] Cleanup after exit with status:2
> 2024-02-01T20:45:43.428575Z 0 [Warning] WSREP: 0.0 (pxc3): State transfer to 2.0 (pxc2) failed: -22 (Invalid argument)
> 2024-02-01T20:45:43.428622Z 0 [ERROR] WSREP: gcs/src/gcs\_group.cpp:gcs\_group\_handle\_join\_msg():811: Will never receive state. Need to abort.
> 2024-02-01T20:45:43.428677Z 0 [Note] WSREP: gcomm: terminating thread
> 2024-02-01T20:45:43.428706Z 0 [Note] WSREP: gcomm: joining thread
> 2024-02-01T20:45:43.428821Z 0 [Note] WSREP: gcomm: closing backend
> 2024-02-01T20:45:43.429438Z 0 [Note] WSREP: Current view of cluster as seen by this node
> view (view\_id(NON\_PRIM,225a4946,188)
> memb {
> d61ae3cb,0
> }
> joined {
> }
> left {
> }
> partitioned {
> 225a4946,0
> 38921ada,0
> f23c4ba4,0
> }
> )
> 2024-02-01T20:45:43.429521Z 0 [Note] WSREP: Current view of cluster as seen by this node
> view ((empty))
> 2024-02-01T20:45:43.429868Z 0 [Note] WSREP: gcomm: closed
> 2024-02-01T20:45:43.429903Z 0 [Note] WSREP: /usr/sbin/mysqld: Terminated.
> Signal 15 (TERM) caught by ps (3.3.16).
> ps:ps/display.c:66: please report this bug
> 2024-02-01T20:45:43.432109Z WSREP\_SST: [ERROR] Removing /var/lib/mysql//.sst/xtrabackup\_galera\_info file due to signal
> 2024-02-01T20:45:43.436861Z WSREP\_SST: [ERROR] Removing file due to signal
---
### Author:  [@matthewb](https://forums.percona.com/u/matthewb)
#### Post date: [February 1, 2024, 10:22pm UTC](https://forums.percona.com/t/cant-reconnect-to-cluster-after-reboot/28141/10 "2024-02-01T22:22:53Z")
> [@Ken\_Scott](#):
>
> 2024-02-01T20:45:32.510815Z WSREP\_SST: [INFO] Proceeding with SST…
> 2024-02-01T20:45:32.559746Z WSREP\_SST: [INFO] …Waiting for SST streaming to complete!
> 2024-02-01T20:45:33.417683Z 0 [Note] WSREP: (d61ae3cb, ‘tcp://0.0.0.0:4567’) turning message relay requesting off
> 2024-02-01T20:45:43.401239Z WSREP\_SST: [ERROR] \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\* FATAL ERROR \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*
> 2024-02-01T20:45:43.403995Z WSREP\_SST: [ERROR] xtrabackup\_checkpoints missing. xtrabackup/SST failed on DONOR. Check DONOR log
On both the donor and the joiner, there should be additional xtrabackup logs that were created as a result of the SST process. Please provide both of those. Since the error occurred within seconds of starting the SST process, I’m leaning towards this being a port issue. Have you confirmed that 4444, 4567, and 4566 are open between all nodes?
---
### Author:  [@Ken\_Scott](https://forums.percona.com/u/Ken_Scott)
#### Post date: [February 2, 2024, 2:11pm UTC](https://forums.percona.com/t/cant-reconnect-to-cluster-after-reboot/28141/11 "2024-02-02T14:11:48Z")
Yes all ports are open. This issue starts on nodes that had been running for 3 years and only occurred after rebooting.
---
### Author:  [@Ken\_Scott](https://forums.percona.com/u/Ken_Scott)
#### Post date: [February 2, 2024, 2:13pm UTC](https://forums.percona.com/t/cant-reconnect-to-cluster-after-reboot/28141/12 "2024-02-02T14:13:55Z")
> [@matthewb](#):
>
> additional xtrabackup logs
I’m not find any addition logs, are they are stored in a different location?
---
### Author:  [@Ken\_Scott](https://forums.percona.com/u/Ken_Scott)
#### Post date: [February 2, 2024, 2:28pm UTC](https://forums.percona.com/t/cant-reconnect-to-cluster-after-reboot/28141/13 "2024-02-02T14:28:25Z")
requesting node:
> 2024-02-02T14:27:06.968274Z 0 [Note] WSREP: Initiating SST/IST transfer on JOINER side (wsrep\_sst\_xtrabackup-v2 --role ‘joiner’ --address ‘172.26.0.12’ --datadir ‘/var/lib/mysql/’ --defaults-file ‘/etc/mysql/my.cnf’ --defaults-group-suffix ‘’ --parent ‘1158781’ --mysqld-version ‘5.7.44-48-57’ ‘’ )
> 2024-02-02T14:27:07.474380Z WSREP\_SST: [INFO] Streaming with xbstream
> 2024-02-02T14:27:07.977430Z 2 [Note] WSREP: Prepared SST/IST request: xtrabackup-v2|172.26.0.12:4444/xtrabackup\_sst//1
> 2024-02-02T14:27:07.977481Z 2 [Note] WSREP: Auto Increment Offset/Increment re-align with cluster membership change (Offset: 1 → 2) (Increment: 1 → 4)
> 2024-02-02T14:27:07.977545Z 2 [Note] WSREP: wsrep\_notify\_cmd is not defined, skipping notification.
> 2024-02-02T14:27:07.977595Z 2 [Note] WSREP: Assign initial position for certification: 7155604377, protocol version: 4
> 2024-02-02T14:27:07.977722Z 0 [Note] WSREP: Service thread queue flushed.
> 2024-02-02T14:27:07.977783Z 2 [Note] WSREP: Check if state gap can be serviced using IST
> 2024-02-02T14:27:07.977827Z 2 [Note] WSREP: Local UUID: fa7ffb4b-c1d6-11ee-a672-fe91ed2bc9b3 != Group UUID: 3a0118b7-6c7b-11eb-935d-5fcc5d08be87
> 2024-02-02T14:27:07.977896Z 2 [Note] WSREP: State gap can’t be serviced using IST. Switching to SST
> 2024-02-02T14:27:07.977911Z 2 [Note] WSREP: Failed to prepare for incremental state transfer: Local state UUID (fa7ffb4b-c1d6-11ee-a672-fe91ed2bc9b3) does not match group state UUID (3a0118b7-6c7b-11eb-935d-5fcc5d08be87): 1 (Operation not permitted)
> at galera/src/replicator\_str.cpp:prepare\_for\_IST():518. IST will be unavailable.
> 2024-02-02T14:27:07.979327Z 0 [Note] WSREP: Member 1.0 (pxc2) requested state transfer from ‘_any_’. Selected 0.0 (pxc3)(SYNCED) as donor.
> 2024-02-02T14:27:07.979366Z 0 [Note] WSREP: Shifting PRIMARY → JOINER (TO: 7155604387)
> 2024-02-02T14:27:07.979412Z 2 [Note] WSREP: Requesting state transfer: success, donor: 0
> 2024-02-02T14:27:07.979457Z 2 [Note] WSREP: GCache history reset: fa7ffb4b-c1d6-11ee-a672-fe91ed2bc9b3:0 → 3a0118b7-6c7b-11eb-935d-5fcc5d08be87:7155604377
> 2024-02-02T14:27:08.616596Z WSREP\_SST: [INFO] Streaming with xbstream
> 2024-02-02T14:27:08.626501Z WSREP\_SST: [INFO] Proceeding with SST…
> 2024-02-02T14:27:08.674417Z WSREP\_SST: [INFO] …Waiting for SST streaming to complete!
> 2024-02-02T14:27:09.466921Z 0 [Note] WSREP: (23e88f89, ‘tcp://0.0.0.0:4567’) turning message relay requesting off
> 2024-02-02T14:27:19.530858Z WSREP\_SST: [ERROR] \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\* FATAL ERROR \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*
> 2024-02-02T14:27:19.533523Z WSREP\_SST: [ERROR] xtrabackup\_checkpoints missing. xtrabackup/SST failed on DONOR. Check DONOR log
> 2024-02-02T14:27:19.536076Z WSREP\_SST: [ERROR] \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*
> 2024-02-02T14:27:19.538683Z 0 [Warning] WSREP: 0.0 (pxc3): State transfer to 1.0 (pxc2) failed: -22 (Invalid argument)
> 2024-02-02T14:27:19.538770Z 0 [ERROR] WSREP: gcs/src/gcs\_group.cpp:gcs\_group\_handle\_join\_msg():811: Will never receive state. Need to abort.
> 2024-02-02T14:27:19.538800Z 0 [Note] WSREP: gcomm: terminating thread
> 2024-02-02T14:27:19.538834Z 0 [Note] WSREP: gcomm: joining thread
> 2024-02-02T14:27:19.538995Z 0 [Note] WSREP: gcomm: closing backend
> 2024-02-02T14:27:19.539639Z 0 [Note] WSREP: Current view of cluster as seen by this node
> view (view\_id(NON\_PRIM,225a4946,192)
> memb {
> 23e88f89,0
> }
> joined {
> }
> left {
> }
> partitioned {
> 225a4946,0
> 38921ada,0
> f23c4ba4,0
> }
> )
> 2024-02-02T14:27:19.539728Z 0 [Note] WSREP: Current view of cluster as seen by this node
> view ((empty))
> 2024-02-02T14:27:19.538984Z WSREP\_SST: [ERROR] Cleanup after exit with status:2
> 2024-02-02T14:27:19.540030Z 0 [Note] WSREP: gcomm: closed
> 2024-02-02T14:27:19.540065Z 0 [Note] WSREP: /usr/sbin/mysqld: Terminated.
> 2024-02-02T14:27:19.544828Z WSREP\_SST: [ERROR] Removing /var/lib/mysql//.sst/xtrabackup\_galera\_info file due to signal
> 2024-02-02T14:27:19.550057Z WSREP\_SST: [ERROR] Removing file due to signal
Doner Node
> 2024-02-02T14:27:07.980161Z 0 [Note] WSREP: Member 1.0 (pxc2) requested state transfer from ‘_any_’. Selected 0.0 (pxc3)(SYNCED) as donor.
> 2024-02-02T14:27:09.138683Z 0 [Note] WSREP: (f23c4ba4, ‘tcp://0.0.0.0:4567’) turning message relay requesting off
> 2024-02-02T14:27:19.539501Z 0 [Warning] WSREP: 0.0 (pxc3): State transfer to 1.0 (pxc2) failed: -22 (Invalid argument)
> 2024-02-02T14:27:19.540970Z 0 [Note] WSREP: Member 0.0 (pxc3) synced with group.
> 2024-02-02T14:27:19.540997Z 0 [Note] WSREP: declaring 225a4946 at tcp://172.26.0.11:4567 stable
> 2024-02-02T14:27:19.541034Z 0 [Note] WSREP: declaring 38921ada at tcp://172.26.0.9:4567 stable
> 2024-02-02T14:27:19.541048Z 0 [Note] WSREP: forgetting 23e88f89 (tcp://172.26.0.12:4567)
> 2024-02-02T14:27:19.541602Z 0 [Note] WSREP: Node 225a4946 state primary
> 2024-02-02T14:27:19.541896Z 0 [Note] WSREP: Current view of cluster as seen by this node
> view (view\_id(PRIM,225a4946,193)
> memb {
> 225a4946,0
> 38921ada,0
> f23c4ba4,0
> }
> joined {
> }
> left {
> }
> partitioned {
> 23e88f89,0
> }
> )
> 2024-02-02T14:27:19.541915Z 0 [Note] WSREP: Save the discovered primary-component to disk
> 2024-02-02T14:27:19.542449Z 0 [Note] WSREP: forgetting 23e88f89 (tcp://172.26.0.12:4567)
> 2024-02-02T14:27:19.542532Z 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my\_idx = 2, memb\_num = 3
> 2024-02-02T14:27:19.542571Z 0 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
> 2024-02-02T14:27:19.543136Z 0 [Note] WSREP: STATE EXCHANGE: sent state msg: 2c00995b-c1d7-11ee-bda8-5aee0bbb8b41
> 2024-02-02T14:27:19.543379Z 0 [Note] WSREP: STATE EXCHANGE: got state msg: 2c00995b-c1d7-11ee-bda8-5aee0bbb8b41 from 0 (pxc3)
> 2024-02-02T14:27:19.543394Z 0 [Note] WSREP: STATE EXCHANGE: got state msg: 2c00995b-c1d7-11ee-bda8-5aee0bbb8b41 from 1 (pxc5)
> 2024-02-02T14:27:19.543406Z 0 [Note] WSREP: STATE EXCHANGE: got state msg: 2c00995b-c1d7-11ee-bda8-5aee0bbb8b41 from 2 (pxc6)
> 2024-02-02T14:27:19.543423Z 0 [Note] WSREP: Quorum results:
> version = 6,
> component = PRIMARY,
> conf\_id = 181,
> members = 3/3 (primary/total),
> act\_id = 7155604420,
> last\_appl. = 7155604297,
> protocols = 0/9/3 (gcs/repl/appl),
> group UUID = 3a0118b7-6c7b-11eb-935d-5fcc5d08be87
> 2024-02-02T14:27:19.543437Z 0 [Note] WSREP: Flow-control interval: [173, 173]
> 2024-02-02T14:27:19.543610Z 6 [Note] WSREP: REPL Protocols: 9 (4, 2)
> 2024-02-02T14:27:19.543646Z 6 [Note] WSREP: REPL Protocols: 9 (4, 2)
> 2024-02-02T14:27:19.543665Z 6 [Note] WSREP: New cluster view: global state: 3a0118b7-6c7b-11eb-935d-5fcc5d08be87:7155604420, view# 182: Primary, number of nodes: 3, my index: 2, protocol version 3
> 2024-02-02T14:27:19.543674Z 6 [Note] WSREP: Setting wsrep\_ready to true
> 2024-02-02T14:27:19.543684Z 6 [Note] WSREP: Auto Increment Offset/Increment re-align with cluster membership change (Offset: 4 → 3) (Increment: 4 → 3)
> 2024-02-02T14:27:19.543784Z 6 [Note] WSREP: wsrep\_notify\_cmd is not defined, skipping notification.
> 2024-02-02T14:27:19.545839Z 6 [Note] WSREP: Assign initial position for certification: 7155604420, protocol version: 4
> 2024-02-02T14:27:19.545955Z 0 [Note] WSREP: Service thread queue flushed.
> 2024-02-02T14:27:24.641379Z 0 [Note] WSREP: cleaning up 23e88f89 (tcp://172.26.0.12:4567)
---
### Author:  [@matthewb](https://forums.percona.com/u/matthewb)
#### Post date: [February 3, 2024, 9:33pm UTC](https://forums.percona.com/t/cant-reconnect-to-cluster-after-reboot/28141/14 "2024-02-03T21:33:20Z")
> [@Ken\_Scott](#):
>
> 2024-02-02T14:27:19.539501Z 0 [Warning] WSREP: 0.0 (pxc3): State transfer to 1.0 (pxc2) failed: -22 (Invalid argument)
We need to see the error logs of the xtrabackup process. They are inside the $datadir of MySQL on both sides.
---
### Author:  [@Ken\_Scott](https://forums.percona.com/u/Ken_Scott)
#### Post date: [February 5, 2024, 2:40pm UTC](https://forums.percona.com/t/cant-reconnect-to-cluster-after-reboot/28141/15 "2024-02-05T14:40:21Z")
Unfortunately there are no xtrabackup logs on the requesting server and on the doner the logs are 3 years old and include no updated info.
---
### Author:  [@matthewb](https://forums.percona.com/u/matthewb)
#### Post date: [February 5, 2024, 4:27pm UTC](https://forums.percona.com/t/cant-reconnect-to-cluster-after-reboot/28141/16 "2024-02-05T16:27:41Z")
I suggest that you perform a manual SST by taking a xtrabackup backup of node1 (use `--galera-info`), transfer to node2, prepare as usual, recreate the grastat.dat, and start the node. It should connect and IST from node1.