Cannot for the life of me, add a node to my current Cluster.

jreiners · December 30, 2020, 4:21pm

We currently have a 3 node cluster set up, and it is working great, I would like to add another 2 nodes to our setup, but it consistently fails. The Percona setup uses 2 sites, locally, we currently have 2 nodes, and our colocation we have a single node (trying to add more) the current node at the colo has been running for 3 months no issue.

Just wondering if ANYONE has an idea, all nodes have been updated to the same version software, selinux is off, firewalls are all off, can telnet to correct ports. This one has me pulling my hair out. I am going to try to add another node to my location here and see if the same thing happens, but I really need to have these other nodes set up and functioning for the new website rollout.

Thanks in advance!

http://pastebin.com/c8WKZ1Ar

marko_s · December 30, 2020, 4:30pm

Are you certain you are running exactly the same versions across all nodes?
5.30 and 5.33 use different versions of xtrabackup for SST and different net streaming pipes (nc and socat, respectively).

From what I’m seeing in logs: [LIST=1]
[*]xb_stream_read_chunk(): wrong chunk magic at offset 0x0.
[/LIST]
The issue is described here:

http://www.percona.com/forums/questions-discussions/percona-xtradb-cluster/11674-upgrade-from-5-5-31-5-5-33-issue

If you pull latest binaries from repos without upgrading existing nodes first, it’s quite possible you ran into 5.30/5.33 SST incompatibility.

Good luck!

jreiners · December 30, 2020, 4:30pm

Thanks! I just worry about upgrading the nudes, best bet is the following? (on the operating nodes)

service mysql stop

yum upgrade Percona* percona*

reboot

does this seem like the best way to go?

jreiners · December 30, 2020, 4:30pm

Scratch that , I did a snapshot of the VM, tried the previous steps, no go, still does the same thing, does this mean I need to shut down and update the whole cluster? I was hoping I could do a rolling upgrade. is this possible?

marko_s · December 30, 2020, 4:30pm

Rolling upgrade is certainly possible, but you need to gather some info first.

Try this:

Check the exact version of all cluster nodes.
Check the exact version on the node you are trying to upgrade (if its 5.5.34 you’re good to go with point 4., if not try to upgrade to that version).
Check if socat is available on all systems.
If you use 5.5.33 on old nodes use wsrep_sst_method =xtrabackup-v2 and if below that version use wsrep_sst_method =xtrabackup on node you are trying to upgrade.

Check this out:
http://www.percona.com/doc/percona-xtradb-cluster/manual/xtrabackup_sst.html

jreiners · December 30, 2020, 4:30pm

Thanks for the info Marko_s!

Do you use [COLOR=#252C2F]wsrep_sst_method =xtrabackup-v2 on the node wanting to join or on the donor node, or both? socat is already installed on all nodes.

Current running node version (Production):
Installed Packages:
Percona-Server-shared-51.x86_64 5.1.72-rel14.10.597.rhel6 @percona
Percona-XtraDB-Cluster-client.x86_64 1:5.5.34-23.7.6.565.rhel6 @percona
Percona-XtraDB-Cluster-galera.x86_64 2.8-1.162.rhel6 @percona
Percona-XtraDB-Cluster-server.x86_64 1:5.5.33-23.7.6.495.rhel6 @percona
Percona-XtraDB-Cluster-shared.x86_64 1:5.5.34-23.7.6.565.rhel6 @percona
percona-release.x86_64 0.0-1 installed
percona-toolkit.noarch 2.2.5-2 @percona
percona-xtrabackup-20.x86_64 2.0.8-587.rhel6 @percona

New nodes I am trying to add have the following versions: (refuses to join existing cluster)
Installed Packages:
Percona-XtraDB-Cluster-client.x86_64 1:5.5.34-23.7.6.565.rhel6 @percona
Percona-XtraDB-Cluster-debuginfo.x86_64 1:5.5.34-23.7.6.565.rhel6 @percona
Percona-XtraDB-Cluster-devel.x86_64 1:5.5.34-23.7.6.565.rhel6 @percona
Percona-XtraDB-Cluster-galera.x86_64 2.8-1.162.rhel6 @percona
Percona-XtraDB-Cluster-galera-debuginfo.x86_64 2.8-1.162.rhel6 @percona
Percona-XtraDB-Cluster-server.x86_64 1:5.5.34-23.7.6.565.rhel6 @percona
Percona-XtraDB-Cluster-shared.x86_64 1:5.5.34-23.7.6.565.rhel6 @percona
Percona-XtraDB-Cluster-test.x86_64 1:5.5.34-23.7.6.565.rhel6 @percona
percona-xtrabackup-20.x86_64 2.0.8-587.rhel6 @percona

Updated nodes (broken after yum update Percona*, will not join)
Installed Packages:
Percona-Server-shared-compat.x86_64 5.5.34-rel32.0.591.rhel6 @percona
Percona-XtraDB-Cluster-client.x86_64 1:5.5.34-23.7.6.565.rhel6 @percona
Percona-XtraDB-Cluster-galera.x86_64 2.8-1.162.rhel6 @percona
Percona-XtraDB-Cluster-server.x86_64 1:5.5.34-23.7.6.565.rhel6 @percona
Percona-XtraDB-Cluster-shared.x86_64 1:5.5.34-23.7.6.565.rhel6 @percona
percona-release.x86_64 0.0-1 installed
percona-xtrabackup.x86_64 2.1.5-680.rhel6 @percona

I cannot get the new or updated notes to join the cluster, error keeps coming up like “Doesn’t look like a tar file” and “no route to host” when I can for sure see and talk to the nodes no problem. I also created symlinks using “ln -s /usr/bin/wsrep_sst_xtrabackup /usr/bin/wsrep_sst_xtrabackup-v2” on the production nodes on older version. Any help would be appreciated. Most of the time the sst dies, and doesnt kill socat, so I have to killall -9 socat to give it another go.

jreiners · December 30, 2020, 4:30pm

When trying to join recently yum updated node I get the following errors.


131118 14:29:55 [Note] WSREP: New cluster view: global state: fbab8da4-2a10-11e3-a6fb-de53a3b76de1:66555457, view# 474: Primary, number of nodes: 5, my index: 0, protocol version 2
131118 14:29:55 [Warning] WSREP: Gap in state sequence. Need state transfer.
131118 14:29:57 [Note] WSREP: Running: 'wsrep_sst_xtrabackup-v2 --role 'joiner' --address '192.168.200.130' --auth 'sstuser:s3cret' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --parent '5059''
sh: wsrep_sst_xtrabackup-v2: command not found
131118 14:29:57 [ERROR] WSREP: Failed to read 'ready <addr>' from: wsrep_sst_xtrabackup-v2 --role 'joiner' --address '192.168.200.130' --auth 'sstuser:s3cret' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --parent '5059'
Read: '(null)'
131118 14:29:57 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'joiner' --address '192.168.200.130' --auth 'sstuser:s3cret' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --parent '5059': 2 (No such file or directory)
131118 14:29:57 [ERROR] WSREP: Failed to prepare for 'xtrabackup-v2' SST. Unrecoverable.
131118 14:29:57 [ERROR] Aborting

marko_s · December 30, 2020, 4:30pm

All right, correct me if I’m wrong, you’re running:

Donor:
Percona-XtraDB-Cluster-server.x86_64 1:5.5.33-23.7.6.495.rhel6 @percona

Joiner:
Percona-XtraDB-Cluster-server.x86_64 1:5.5.34-23.7.6.565.rhel6 @percona

If that’s the case, set wsrep_sst_method = xtrabackup-v2 on JOINER, and wsrep_sst_method = xtrabackup on DONOR.

Also, maybe someone else can pipe in concerning what’s exactly in which RPM since you have 5.33 server and client/shared libs v. 5.34 RPM’s on DONOR.
From above, you don’t need symlink for xtrabackup-v2 on donor, use xtrabackup on donor and v2 on joiner.

Now, the error you posted seems like xtrabackup-v2 does not exist or is not in the path:

sh: wsrep_sst_xtrabackup-v2: command not found

If the node you are trying to bring up (joiner) is server 5.34 it should have this binary somewhere in bin directory (again, I’m running stuff from /opt/mysql/bin and the binary is there allright). Check that the binary exists on JOINER and that the my.cnf has correct paths or it is in sys path.

You can also try running yum update on [COLOR=#252C2F]percona-xtrabackup on new nodes (again, dunno what goes in what package).

jreiners · December 30, 2020, 4:30pm

Thanks again!

You are correct with the versions, I still, cannot add another node running new version to the cluster, In the repo I am seeing both “percona-xtrabackup-20” and “percona-xtrabackup” Do you have any idea which one I am supposed to have installed? I assume percona-xtrabackup-20 is 2.0

I still get the same errors when I try to add it in. I CAN clone the older VMs and they add in fine, just not the new updated nodes.

marko_s · December 30, 2020, 4:30pm

Sorry, no idea about packages, however I checked and what you really need is the binary named wsrep_sst_xtrabackup-v2 that comes in with PXC install.

Try on the 5.34 joiner:

find / -name wsrep_sst_xtrabackup*

See if it yields any results. Judging by the log mysql cannot find the binary…


sh: wsrep_sst_xtrabackup-v2: command not found