update pxc node from 5.6.16 to 5.6.19

MySQL & MariaDB Percona XtraDB Cluster 5.x
1

on RHEL 6
Percona-XtraDB-Cluster-server-56-5.6.15-25.5.759.rhel6.x86_64
(with Percona-XtraDB-Cluster-galera-3-3.5-1.216.rhel6.x86_64)
is running a 3 node cluster fine

upgrading one node to
Percona-XtraDB-Cluster-shared-56-5.6.19-25.6.824.el6.x86_64
(and to Percona-XtraDB-Cluster-galera-3-3.6-1.3190.rhel6.x86_6)
breaks it, SST is working but then:

2014-08-11 16:25:45 32720 [Note] WSREP: Member 2.0 (xdb8071) requested state transfer from ‘xdb8069’. Selected 1.0 (xdb8069)(SYNCED) as donor.
2014-08-11 16:25:45 32720 [Note] WSREP: Shifting PRIMARY → JOINER (TO: 6558)
2014-08-11 16:25:45 32720 [Note] WSREP: Requesting state transfer: success, donor: 1
WSREP_SST: [INFO] Proceeding with SST (20140811 16:25:45.992)
WSREP_SST: [INFO] Cleaning the existing datadir (20140811 16:25:45.995)
removed /data1/mysql/data/gvwstate.dat' WSREP_SST: [INFO] Cleaning the binlog directory /data/mysql/logs as well (20140811 16:25:46.017) rm: cannot remove /data/mysql/logs/.index’: No such file or directory
WSREP_SST: [INFO] Evaluating socat -u TCP-LISTEN:4444,reuseaddr stdio | xbstream -x; RC=( ${PIPESTATUS[@]} ) (20140811 16:25:46.024)
2014-08-11 16:26:26 32720 [Note] WSREP: 1.0 (xdb8069): State transfer to 2.0 (xdb8071) complete.
2014-08-11 16:26:26 32720 [Note] WSREP: Member 1.0 (xdb8069) synced with group.
ls: cannot access mysql-binlog-8071.: No such file or directory
WSREP_SST: [INFO] Preparing the backup at /data1/mysql/data/ (20140811 16:26:26.548)
WSREP_SST: [INFO] Evaluating innobackupex --no-version-check --apply-log $rebuildcmd ${DATA} &>${DATA}/innobackup.prepare.log (20140811 16:26:26.551)
WSREP_SST: [INFO] Total time on joiner: 0 seconds (20140811 16:26:35.265)
WSREP_SST: [INFO] Removing the sst_in_progress file (20140811 16:26:35.268)
2014-08-11 16:26:35 32720 [Note] WSREP: SST complete, seqno: 6558
2014-08-11 16:26:35 32720 [Note] Plugin ‘FEDERATED’ is disabled.
2014-08-11 16:26:35 32720 [Note] InnoDB: Using atomics to ref count buffer pool pages
2014-08-11 16:26:35 32720 [Note] InnoDB: The InnoDB memory heap is disabled
2014-08-11 16:26:35 32720 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2014-08-11 16:26:35 32720 [Note] InnoDB: Compressed tables use zlib 1.2.3
2014-08-11 16:26:35 32720 [Note] InnoDB: Using Linux native AIO
2014-08-11 16:26:35 32720 [Note] InnoDB: Using CPU crc32 instructions
2014-08-11 16:26:35 32720 [Note] InnoDB: Initializing buffer pool, size = 39.1G
2014-08-11 16:26:37 32720 [Note] InnoDB: Completed initialization of buffer pool
2014-08-11 16:26:37 32720 [Note] InnoDB: Highest supported file format is Barracuda.
2014-08-11 16:26:38 32720 [Note] InnoDB: 128 rollback segment(s) are active.
2014-08-11 16:26:38 32720 [Note] InnoDB: Waiting for purge to start
2014-08-11 16:26:38 32720 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.19-67.0 started; log sequence number 3816493669
2014-08-11 16:26:38 32720 [ERROR] Aborting

2014-08-11 16:26:40 32720 [Note] WSREP: Closing send monitor…
2014-08-11 16:26:40 32720 [Note] WSREP: Closed send monitor.
…
2014-08-11 16:26:44 32720 [Note] InnoDB: Shutdown completed; log sequence number 3816496970

log of the donor node:

2014-08-11 16:26:26 3587 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
WSREP_SST: [INFO] Total time on donor: 0 seconds (20140811 16:26:26.530)
WSREP_SST: [INFO] Cleaning up temporary directories (20140811 16:26:26.536)
2014-08-11 16:26:40 3587 [Note] WSREP: declaring 409fe6e9-0da2-11e4-a8dc-f6c52645ad47 stable
2014-08-11 16:26:40 3587 [Note] WSREP: (5df674bc-0da2-11e4-8264-afe0e5387779, ‘tcp://0.0.0.0:4567’) address ‘tcp://10.64.218.40:4567’ pointing to uuid 5df674bc-0da2-11e4-8264-afe0e5387779 is blacklisted, s
kipping
2014-08-11 16:26:40 3587 [Note] WSREP: (5df674bc-0da2-11e4-8264-afe0e5387779, ‘tcp://0.0.0.0:4567’) address ‘tcp://10.64.218.40:4567’ pointing to uuid 5df674bc-0da2-11e4-8264-afe0e5387779 is blacklisted, s
kipping
2014-08-11 16:26:40 3587 [Note] WSREP: (5df674bc-0da2-11e4-8264-afe0e5387779, ‘tcp://0.0.0.0:4567’) turning message relay requesting on, nonlive peers: tcp://10.64.218.42:4567
2014-08-11 16:26:40 3587 [Note] WSREP: Node 409fe6e9-0da2-11e4-a8dc-f6c52645ad47 state prim
2014-08-11 16:26:40 3587 [Note] WSREP: view(view_id(PRIM,409fe6e9-0da2-11e4-a8dc-f6c52645ad47,34) memb {
409fe6e9-0da2-11e4-a8dc-f6c52645ad47,0
5df674bc-0da2-11e4-8264-afe0e5387779,0
} joined {
} left {
} partitioned {
5eb5d970-2163-11e4-ad72-7bd016470afd,0
})
2014-08-11 16:26:40 3587 [Note] WSREP: forgetting 5eb5d970-2163-11e4-ad72-7bd016470afd (tcp://10.64.218.42:4567)
2014-08-11 16:26:40 3587 [Note] WSREP: (5df674bc-0da2-11e4-8264-afe0e5387779, ‘tcp://0.0.0.0:4567’) address ‘tcp://10.64.218.40:4567’ pointing to uuid 5df674bc-0da2-11e4-8264-afe0e5387779 is blacklisted, s
kipping
2014-08-11 16:26:40 3587 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 2
2014-08-11 16:26:40 3587 [Note] WSREP: (5df674bc-0da2-11e4-8264-afe0e5387779, ‘tcp://0.0.0.0:4567’) turning message relay requesting off
2014-08-11 16:26:40 3587 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
2014-08-11 16:26:40 3587 [Note] WSREP: STATE EXCHANGE: sent state msg: 82731576-2163-11e4-9a53-87f667275baa
2014-08-11 16:26:40 3587 [Note] WSREP: STATE EXCHANGE: got state msg: 82731576-2163-11e4-9a53-87f667275baa from 0 (xdb8070)
2014-08-11 16:26:40 3587 [Note] WSREP: STATE EXCHANGE: got state msg: 82731576-2163-11e4-9a53-87f667275baa from 1 (xdb8069)
2014-08-11 16:26:40 3587 [Note] WSREP: Quorum results:
version = 3,
component = PRIMARY,
conf_id = 32,
members = 2/2 (joined/total),
act_id = 6558,
last_appl. = 6527,
protocols = 0/5/2 (gcs/repl/appl),
group UUID = fd641b3d-0d07-11e4-bb59-bfa063c9be5f
2014-08-11 16:26:40 3587 [Note] WSREP: Flow-control interval: [23, 23]
2014-08-11 16:26:40 3587 [Note] WSREP: New cluster view: global state: fd641b3d-0d07-11e4-bb59-bfa063c9be5f:6558, view# 33: Primary, number of nodes: 2, my index: 1, protocol version 2
2014-08-11 16:26:40 3587 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2014-08-11 16:26:40 3587 [Note] WSREP: REPL Protocols: 5 (3, 1)
2014-08-11 16:26:40 3587 [Note] WSREP: Service thread queue flushed.
2014-08-11 16:26:40 3587 [Note] WSREP: Assign initial position for certification: 6558, protocol version: 3
2014-08-11 16:26:40 3587 [Note] WSREP: Service thread queue flushed.
2014-08-11 16:26:40 3587 [Warning] WSREP: Releasing seqno 6558 before 6559 was assigned.
2014-08-11 16:26:45 3587 [Note] WSREP: cleaning up 5eb5d970-2163-11e4-ad72-7bd016470afd (tcp://10.64.218.42:4567)

yum downgrade to 5.6.15. and node is up again.

I tried:
rm -rf in the datadir to force an SST, didn’t help.

Have I done it wrong? Is there a need to upgrade galera first on all node or something like this, or is 5.6.19 broken?

2

Hello
I have set up a second Cluster on different maschines, 5.6.19 installed.
I copied a backup (made with xtrabackup) from the 5.6.15 Cluster and tried to bootstrap that copy

-------------error.log
140814 17:42:41 mysqld_safe Starting mysqld daemon with databases from /data1/mysql/data
140814 17:42:41 mysqld_safe Skipping wsrep-recover for empty datadir: /data1/mysql/data
140814 17:42:41 mysqld_safe Assigning 00000000-0000-0000-0000-000000000000:-1 to wsrep_start_position
2014-08-14 17:42:42 0 [Warning] Using unique option prefix character_set_client instead of character-set-client-handshake is deprecated and will be removed in a future release. Pl
ease use the full name instead.
2014-08-14 17:42:42 0 [Warning] /usr/sbin/mysqld: ignoring option ‘–character-set-client-handshake’ due to invalid value ‘utf8’
2014-08-14 17:42:42 0 [Note] WSREP: wsrep_start_position var submitted: ‘00000000-0000-0000-0000-000000000000:-1’
2014-08-14 17:42:42 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details
).
2014-08-14 17:42:42 24446 [ERROR] Can’t find messagefile ‘/usr/share/mysql/errmsg.sys’
2014-08-14 17:42:42 24446 [Note] WSREP: Read nil XID from storage engines, skipping position init
2014-08-14 17:42:42 24446 [Note] WSREP: wsrep_load(): loading provider library ‘/usr/lib64/libgalera_smm.so’
2014-08-14 17:42:42 24446 [Note] WSREP: wsrep_load(): Galera 3.6(r3a949e6) by Codership Oy <info@codership.com> loaded successfully.
2014-08-14 17:42:42 24446 [Note] WSREP: CRC-32C: using hardware acceleration.
2014-08-14 17:42:42 24446 [Warning] WSREP: Could not open saved state file for reading: /data1/mysql/data//grastate.dat
2014-08-14 17:42:42 24446 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1
2014-08-14 17:42:42 24446 [Note] WSREP: Passing config to GCS: base_host = 10.64.224.70; base_port = 4567; cert.log_conflicts = no; debug = no; evs.inactive_check_period = PT0.5S;
evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.u
ser_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /data1/mysql/data/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /data1/mysql/data//galera.cac
he; gcache.page_size = 128M; gcache.size = 256M; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle
= 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc.chec
ksum = false; pc.ignore_quorum = false; pc.ignore_sb = false; pc.npvo = false; pc.recovery = 1; pc.version = 0; pc.wait_prim = true; pc.wait_prim_timeout = P30S;
2014-08-14 17:42:42 24446 [Note] WSREP: Service thread queue flushed.
2014-08-14 17:42:42 24446 [Note] WSREP: Assign initial position for certification: -1, protocol version: -1
2014-08-14 17:42:42 24446 [Note] WSREP: wsrep_sst_grab()
2014-08-14 17:42:42 24446 [Note] WSREP: Start replication
2014-08-14 17:42:42 24446 [Note] WSREP: Setting initial position to 00000000-0000-0000-0000-000000000000:-1
2014-08-14 17:42:42 24446 [Note] WSREP: protonet asio version 0
2014-08-14 17:42:42 24446 [Note] WSREP: Using CRC-32C (optimized) for message checksums.
2014-08-14 17:42:42 24446 [Note] WSREP: backend: asio
2014-08-14 17:42:42 24446 [Warning] WSREP: access file(gvwstate.dat) failed(No such file or directory)
2014-08-14 17:42:42 24446 [Note] WSREP: restore pc from disk failed
2014-08-14 17:42:42 24446 [Note] WSREP: GMCast version 0
2014-08-14 17:42:42 24446 [Note] WSREP: (a0d899d6, ‘tcp://0.0.0.0:4567’) listening at tcp://0.0.0.0:4567
2014-08-14 17:42:42 24446 [Note] WSREP: (a0d899d6, ‘tcp://0.0.0.0:4567’) multicast: , ttl: 1
2014-08-14 17:42:42 24446 [Note] WSREP: EVS version 0
2014-08-14 17:42:42 24446 [Note] WSREP: PC version 0
2014-08-14 17:42:42 24446 [Note] WSREP: gcomm: connecting to group ‘stage_sp1_xdb_1’, peer ‘’
2014-08-14 17:42:42 24446 [Note] WSREP: start_prim is enabled, turn off pc_recovery
2014-08-14 17:42:42 24446 [Note] WSREP: Node a0d899d6 state prim
2014-08-14 17:42:42 24446 [Note] WSREP: save pc into disk
2014-08-14 17:42:42 24446 [Note] WSREP: gcomm: connected
2014-08-14 17:42:42 24446 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636
2014-08-14 17:42:42 24446 [Note] WSREP: Shifting CLOSED → OPEN (TO: 0)
2014-08-14 17:42:42 24446 [Note] WSREP: Opened channel ‘stage_sp1_xdb_1’
2014-08-14 17:42:42 24446 [Note] WSREP: Waiting for SST to complete.
2014-08-14 17:42:42 24446 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 1
2014-08-14 17:42:42 24446 [Note] WSREP: Starting new group from scratch: a0d977c2-23c9-11e4-8b41-b707239bd1c3
2014-08-14 17:42:42 24446 [Note] WSREP: STATE_EXCHANGE: sent state UUID: a0d989bb-23c9-11e4-8481-d33e3e29b875
2014-08-14 17:42:42 24446 [Note] WSREP: STATE EXCHANGE: sent state msg: a0d989bb-23c9-11e4-8481-d33e3e29b875
2014-08-14 17:42:42 24446 [Note] WSREP: STATE EXCHANGE: got state msg: a0d989bb-23c9-11e4-8481-d33e3e29b875 from 0 (xdb8284)
2014-08-14 17:42:42 24446 [Note] WSREP: Quorum results:
version = 3,
component = PRIMARY,
conf_id = 0,
members = 1/1 (joined/total),
act_id = 0,
last_appl. = -1,
protocols = 0/6/3 (gcs/repl/appl),
group UUID = a0d977c2-23c9-11e4-8b41-b707239bd1c3
2014-08-14 17:42:42 24446 [Note] WSREP: Flow-control interval: [16, 16]
2014-08-14 17:42:42 24446 [Note] WSREP: Restored state OPEN → JOINED (0)
2014-08-14 17:42:42 24446 [Note] WSREP: Member 0.0 (xdb8284) synced with group.
2014-08-14 17:42:42 24446 [Note] WSREP: Shifting JOINED → SYNCED (TO: 0)
2014-08-14 17:42:42 24446 [Note] WSREP: New cluster view: global state: a0d977c2-23c9-11e4-8b41-b707239bd1c3:0, view# 1: Primary, number of nodes: 1, my index: 0, protocol version 3
2014-08-14 17:42:42 24446 [Note] WSREP: SST complete, seqno: 0
2014-08-14 17:42:42 24446 [Note] Plugin ‘FEDERATED’ is disabled.
2014-08-14 17:42:42 24446 [Note] InnoDB: Using atomics to ref count buffer pool pages
2014-08-14 17:42:42 24446 [Note] InnoDB: The InnoDB memory heap is disabled
2014-08-14 17:42:42 24446 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2014-08-14 17:42:42 24446 [Note] InnoDB: Compressed tables use zlib 1.2.3
2014-08-14 17:42:42 24446 [Note] InnoDB: Using Linux native AIO
2014-08-14 17:42:42 24446 [Note] InnoDB: Using CPU crc32 instructions
2014-08-14 17:42:42 24446 [Note] InnoDB: Initializing buffer pool, size = 7.8G
2014-08-14 17:42:42 24446 [Note] InnoDB: Completed initialization of buffer pool
2014-08-14 17:42:42 24446 [Note] InnoDB: Highest supported file format is Barracuda.
2014-08-14 17:42:43 24446 [Note] InnoDB: 128 rollback segment(s) are active.
2014-08-14 17:42:43 24446 [Note] InnoDB: Waiting for purge to start
2014-08-14 17:42:43 24446 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.19-67.0 started; log sequence number 6281825897
2014-08-14 17:42:43 24446 [ERROR] Aborting

yum downgrade Percona-XtraDB-Cluster-client-56-5.6.15 Percona-XtraDB-Cluster-galera-3-3.5 Percona-XtraDB-Cluster-shared-56-5.6.15 Percona-XtraDB-Cluster-server-56-5.6.15

/etc/init.d/mysql start
ERROR! MySQL (Percona XtraDB Cluster) is not running, but lock file (/var/lock/subsys/mysql) exists
Starting MySQL (Percona XtraDB Cluster)… SUCCESS!

I didn’t searched for the changelog yet, did I miss something about 5.6.19?

I first tried mysql_install_db, but this got this also:
2014-08-14 17:42:43 24446 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.19-67.0 started; log sequence number 6281825897
2014-08-14 17:42:43 24446 [ERROR] Aborting

Seems like a bug to me, but as this version is out for weeks now, I wonder I can’t find anything about it.

3 
2014-08-14 17:42:42 24446 [ERROR] Can't find messagefile '/usr/share/mysql/errmsg.sys'

is the key here. Removing hardcoded lc-messages-dir path from my.cnf will help.

4

I have the same error

InnoDB: Percona XtraDB (http://www.percona.com) 5.6.29-76.2 started; log sequence number 1625987
2016-03-25 14:22:48 31392 [ERROR] Aborting

That’s the last error message before server shutdown and was never able to open
this is a brand new install of percona 5.6 on a CentOS 7 box
with a minimum /etc/my.cnf
What’s going on here - why doesn’t it want to start
and why not more errors in error.log
what to do ?