Upgrade from 5.5.31 -> 5.5.33 issue

[marko_s]

Hello,

We decided to try upgrading one node to latest stable.

The OS in question is CentOS 6.3 x64.

It seems that streaming of SST has changed from using nc to socat? Obviously it failed because I had to recompile and install socat manually.
Furthermore, trying to sync 5.5.33 with 5.5.31 yields “xb_stream_read_chunk(): wrong chunk magic at offset 0x0.”

I’m guessing the donor is trying to stream via nc, and joiner is listening on socat?

How to remedy the upgrade? Any way to force nc on joiner? Or is this issue unrelated to socat?
I don’t remember seeing nothing relevant in release notes.

Regards,
Marko

[tomksoft]

Faced the same problem today. Seems like the default stream format was changed from tar to xbstream when doing xtrabackup SST. Because of this 5.5.31 sends tar and 5.5.33 expects xbstream format(and errors with invalid magic, which I believe is a binary header that identifies xbstream).

Good news is it can be configured: http://www.percona.com/doc/percona-x…ackup_sst.html

[sst]
streamfmt=tar

After this change, state transfer completes, however I got stuck just after state is transferred:
WSREP_SST: [INFO] Evaluating nc -dl 4444 | pv -f -i 10 -N joiner | tar xfi - ; RC=( ${PIPESTATUS[@]} ) (20130921 04:47:14.457)

Nothing happens on the donor and the joiner gets stuck. There are no errors. Perhaps it is because I switched from socat to nc (via my.cnf), now I am tring with socat.

Update:
Same happens with socat:

On joiner:

WSREP_SST: [INFO] Proceeding with SST (20130921 05:18:07.838)
WSREP_SST: [INFO] Cleaning the existing datadir (20130921 05:18:07.840)
removed /var/lib/mysql/data/ib_logfile0' removed /var/lib/mysql/data/ib_logfile1’
removed `/var/lib/mysql/data/ibdata1’
WSREP_SST: [INFO] Evaluating socat -u TCP-LISTEN:4444,reuseaddr stdio | pv -f -i 10 -N joiner | tar xfi - ; RC=( ${PIPESTATUS[@]} ) (20130921 05:18:07.846)
130921 5:18:08 [Note] WSREP: 0 (XXXXXXXXXXX): State transfer to 4 (XXXXXXXXXXX) complete.
130921 5:18:08 [Note] WSREP: Member 0 (XXXXXXXXXXX) synced with group.
joiner: 0B 0:00:10 [ 0B/s ] [<=> ]
joiner: 0B 0:00:20 [ 0B/s ] [<=> ]

On donor:
130921 12:18:08 [Note] WSREP: Shifting DONOR/DESYNCED -> JOINED (TO: 408859845)
130921 12:18:08 [Note] WSREP: Member 0 (XXXXXXXXXXX) synced with group.
130921 12:18:08 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 408859850)
130921 12:18:08 [Note] WSREP: Synchronized with group, ready for connections
130921 12:18:08 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.

Looks like joiner is waiting for data that donor is not sending.

[Raghavendra_Prabhu]

@Marko, @Tomksoft, Thanks for reporting this. It has been documented now here: http://www.percona.com/doc/percona-xtradb-cluster/errata.html#incompatibilities and http://www.percona.com/doc/percona-xtradb-cluster/manual/xtrabackup_sst.html Let us know if the workarounds help.

[tomksoft]

Thank you. I went for the 1st workaround of copying sst scripts and it worked really well. I am just slightly sad I did not think about it myself

[marko_s]

Here’s my initial procedure of upgrading the cluster with new version (4/5 nodes done):

1. Copy new binaries to a node selected as donor
2. Force donor node via wsrep_sst_donor
3. Install socat (if missing). Also had to link it in /usr/sbin since it is compiles to /usr/local/bin by default (CentOS 6.3)
4. Our mysql.sock is at /temp not /var/lib. I had to introduce [xtrabackup] group in my.cnf with socket directive at donor node else xtrabackup could not find it. Had no issue with that on previous releases.
5. Switch forcing donor to newly fully upgraded node which will now seed the rest of the cluster
6. Upgrade/SST/IST as necessary forcing fresh 5.33 as donor node
7. Remove wsrep_sst_donor directive

NOTE: I have been able to upgrade other nodes purely via IST once I had setup all the options above correctly and had 5.33 donor node up&running.
We are using binary tar.gz distributions and upgrading the Percona software consisted of moving/renaming binaries/data folders.

Question - Is it necessary to install the following packages alongside binary distro?

Percona-XtraDB-Cluster-shared
Percona-Server-shared-compat
Percona-XtraDB-Cluster-client

As far as I can see, these are mostly MySQL client apps and such.