# MariaDB Galera cluster with xtrabackup issue **URL:** \ **Category:** Other MySQL® Questions\ **Created:** [February 14, 2018, 8:39am UTC](https://forums.percona.com/t/mariadb-galera-cluster-with-xtrabackup-issue/6195 "2018-02-14T08:39:43Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![maksimse](https://avatars.discourse-cdn.com/v4/letter/m/b487fb/32.png) [@maksimse](https://forums.percona.com/u/maksimse)\ **Post date:** [February 14, 2018, 8:39am UTC](https://forums.percona.com/t/mariadb-galera-cluster-with-xtrabackup-issue/6195/1 "2018-02-14T08:39:43Z")
Hello! I have recently installed Galera cluster, but I cannot get up running mariadb on the second node. NODE1 /etc/my.cnf.d/server.cnf: [mysqld] [embedded] [mysqld-5.5] [mariadb] datadir=/var/lib/mysql tmpdir=/tmp/zabbix [mariadb-10.1] optimizer\_switch = ‘index\_condition\_pushdown=off’ wsrep\_on=ON wsrep\_provider=/usr/lib64/galera/libgalera\_smm.so wsrep\_cluster\_address=“gcomm://10.10.16.1,10.10.16.2” wsrep\_cluster\_name=‘zabbix-cluster’ wsrep\_node\_address=‘10.10.16.1’ wsrep\_node\_name=‘zabbix-db-node1’ wsrep\_sst\_method=xtrabackup binlog\_format=row innodb\_autoinc\_lock\_mode=2 innodb\_doublewrite=1 query\_cache\_size = 0 query\_cache\_type = 0 default\_storage\_engine=InnoDB wsrep\_sst\_auth=“zabbix\_db\_node1:zabbix\_db\_node1” innodb\_flush\_log\_at\_trx\_commit = 2 wsrep\_node\_incoming\_address=‘10.10.16.1’ wsrep\_sst\_receive\_address=‘10.10.16.1’ innodb\_file\_per\_table innodb\_buffer\_pool\_size = 10G innodb\_flush\_method = O\_DIRECT innodb\_log\_buffer\_size = 128M innodb\_lock\_wait\_timeout = 500 innodb\_log\_file\_size = 100M [mariadb-5.5] NODE2 /etc/my.cnf.d/server.cnf: [mysqld] [embedded] [mysqld-5.5] [mariadb] datadir=/var/lib/mysql tmpdir=/tmp/zabbix [mariadb-10.1] optimizer\_switch = ‘index\_condition\_pushdown=off’ wsrep\_on=ON wsrep\_provider=/usr/lib64/galera/libgalera\_smm.so wsrep\_cluster\_address=“gcomm://10.10.16.1,10.10.16.2” wsrep\_cluster\_name=‘zabbix-cluster’ wsrep\_node\_address=‘10.10.16.2’ wsrep\_node\_name=‘zabbix-db-node2’ wsrep\_sst\_method=xtrabackup binlog\_format=row innodb\_doublewrite=1 query\_cache\_size = 0 query\_cache\_type = 0 innodb\_autoinc\_lock\_mode=2 default\_storage\_engine=InnoDB wsrep\_sst\_auth=“zabbix\_db\_node2:zabbix\_db\_node2” innodb\_flush\_log\_at\_trx\_commit = 2 wsrep\_node\_incoming\_address=‘10.10.16.2’ wsrep\_sst\_receive\_address=‘10.10.16.2’ innodb\_file\_per\_table innodb\_buffer\_pool\_size = 10G innodb\_flush\_method = O\_DIRECT innodb\_log\_buffer\_size = 128M innodb\_lock\_wait\_timeout = 500 innodb\_log\_file\_size = 100M [mariadb-5.5] 1. I start galera\_new\_cluster on NODE1. Everything seems to be OK: # systemctl status mariadb ● mariadb.service - MariaDB 10.1.31 database server Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled) Drop-In: /etc/systemd/system/mariadb.service.d └─migrated-from-my.cnf-settings.conf Active: active (running) since Wed 2018-02-14 05:31:09 EST; 8min ago Docs: man:mysqld(8) [https://mariadb.com/kb/en/library/systemd/](https://mariadb.com/kb/en/library/systemd/) Process: 10744 ExecStartPost=/bin/sh -c systemctl unset-environment \_WSREP\_START\_POSITION (code=exited, status=0/SUCCESS) Process: 10694 ExecStartPre=/bin/sh -c [! -e /usr/bin/galera\_recovery] && VAR= || VAR=`/usr/bin/galera_recovery`; [$? -eq 0] && systemctl set-environment \_WSREP\_START\_POSITION=$VAR || exit 1 (code=exited, status=0/SUCCESS) Process: 10691 ExecStartPre=/bin/sh -c systemctl unset-environment \_WSREP\_START\_POSITION (code=exited, status=0/SUCCESS) Main PID: 10711 (mysqld) Status: “Taking your SQL requests now…” CGroup: /system.slice/mariadb.service └─10711 /usr/sbin/mysqld --wsrep-new-cluster Feb 14 05:31:19 zabbix-db-node1 mysqld[10711]: 2018-02-14 5:31:19 139659971651328 [Note] WSREP: Flow-control interval: [16, 16] Feb 14 05:31:19 zabbix-db-node1 mysqld[10711]: 2018-02-14 5:31:19 139659971651328 [Note] WSREP: Trying to continue unpaused monitor Feb 14 05:31:19 zabbix-db-node1 mysqld[10711]: 2018-02-14 5:31:19 139660283120384 [Note] WSREP: New cluster view: global state: ad8e3ac3-116d-11e8-859f-eadc403c6207:1780, view# 3: P…ol version 3 Feb 14 05:31:19 zabbix-db-node1 mysqld[10711]: 2018-02-14 5:31:19 139660283120384 [Note] WSREP: wsrep\_notify\_cmd is not defined, skipping notification. Feb 14 05:31:19 zabbix-db-node1 mysqld[10711]: 2018-02-14 5:31:19 139660283120384 [Note] WSREP: REPL Protocols: 7 (3, 2) Feb 14 05:31:19 zabbix-db-node1 mysqld[10711]: 2018-02-14 5:31:19 139660283120384 [Note] WSREP: Assign initial position for certification: 1780, protocol version: 3 Feb 14 05:31:19 zabbix-db-node1 mysqld[10711]: 2018-02-14 5:31:19 139660030375680 [Note] WSREP: Service thread queue flushed. Feb 14 05:31:23 zabbix-db-node1 mysqld[10711]: 2018-02-14 5:31:23 139659980044032 [Note] WSREP: (2b3ca384, ‘tcp://0.0.0.0:4567’) connection established to 3324f771 tcp://10.10.16.2:4567 Feb 14 05:31:23 zabbix-db-node1 mysqld[10711]: 2018-02-14 5:31:23 139659980044032 [Warning] WSREP: discarding established (time wait) 3324f771 (tcp://10.10.16.2:4567) Feb 14 05:31:24 zabbix-db-node1 mysqld[10711]: 2018-02-14 5:31:24 139659980044032 [Note] WSREP: cleaning up 3324f771 (tcp://10.10.16.2:4567) Hint: Some lines were ellipsized, use -l to show in full. 2. I systemctl start mariadb on NODE2, wait some 30 seconds and get: 3. Then I look into the journal: : Starting MariaDB 10.1.31 database server… Feb 14 12:48:03 zabbix-db-node2 mysqld[3150]: 2018-02-14 12:48:03 139646100506880 [Note] /usr/sbin/mysqld (mysqld 10.1.31-MariaDB) starting as process 3150 … Feb 14 12:48:03 zabbix-db-node2 mysqld[3150]: 2018-02-14 12:48:03 139646100506880 [Note] WSREP: Read nil XID from storage engines, skipping position init Feb 14 12:48:03 zabbix-db-node2 mysqld[3150]: 2018-02-14 12:48:03 139646100506880 [Note] WSREP: wsrep\_load(): loading provider library ‘/usr/lib64/galera/libgalera\_smm.so’ Feb 14 12:48:03 zabbix-db-node2 mysqld[3150]: 2018-02-14 12:48:03 139646100506880 [Note] WSREP: wsrep\_load(): Galera 25.3.22(r3764) by Codership Oy \ loaded successfully. … … … Feb 14 12:48:03 zabbix-db-node2 mysqld[3150]: 2018-02-14 12:48:03 139646100506880 [Note] WSREP: gcomm thread scheduling priority set to other:0 Feb 14 12:48:03 zabbix-db-node2 mysqld[3150]: 2018-02-14 12:48:03 139646100506880 [Warning] WSREP: access file(/var/lib/mysql//gvwstate.dat) failed(No such file or directory) Feb 14 12:48:03 zabbix-db-node2 mysqld[3150]: 2018-02-14 12:48:03 139646100506880 [Note] WSREP: restore pc from disk failed Feb 14 12:48:03 zabbix-db-node2 mysqld[3150]: 2018-02-14 12:48:03 139646100506880 [Note] WSREP: GMCast version 0 Feb 14 12:48:03 zabbix-db-node2 mysqld[3150]: 2018-02-14 12:48:03 139646100506880 [Note] WSREP: (887aad3d, ‘tcp://0.0.0.0:4567’) listening at tcp://0.0.0.0:4567 Feb 14 12:48:03 zabbix-db-node2 mysqld[3150]: 2018-02-14 12:48:03 139646100506880 [Note] WSREP: (887aad3d, ‘tcp://0.0.0.0:4567’) multicast: , ttl: 1 Feb 14 12:48:03 zabbix-db-node2 mysqld[3150]: 2018-02-14 12:48:03 139646100506880 [Note] WSREP: EVS version 0 Feb 14 12:48:03 zabbix-db-node2 mysqld[3150]: 2018-02-14 12:48:03 139646100506880 [Note] WSREP: gcomm: connecting to group ‘zabbix-cluster’, peer ‘10.10.16.1:,10.10.16.2:’ Feb 14 12:48:03 zabbix-db-node2 mysqld[3150]: 2018-02-14 12:48:03 139646100506880 [Note] WSREP: (887aad3d, ‘tcp://0.0.0.0:4567’) connection established to 887aad3d tcp://10.10.16.2:4567 Feb 14 12:48:03 zabbix-db-node2 mysqld[3150]: 2018-02-14 12:48:03 139646100506880 [Warning] WSREP: (887aad3d, ‘tcp://0.0.0.0:4567’) address ‘tcp://10.10.16.2:4567’ points to own listening address, … … … Feb 14 12:48:04 zabbix-db-node2 mysqld[3150]: 2018-02-14 12:48:04 139645799098112 [Note] WSREP: Flow-control interval: [23, 23] Feb 14 12:48:04 zabbix-db-node2 mysqld[3150]: 2018-02-14 12:48:04 139645799098112 [Note] WSREP: Trying to continue unpaused monitor Feb 14 12:48:04 zabbix-db-node2 mysqld[3150]: 2018-02-14 12:48:04 139645799098112 [Note] WSREP: Shifting OPEN → PRIMARY (TO: 2321) Feb 14 12:48:04 zabbix-db-node2 mysqld[3150]: 2018-02-14 12:48:04 139646100187904 [Note] WSREP: State transfer required: Feb 14 12:48:04 zabbix-db-node2 mysqld[3150]: Group state: ad8e3ac3-116d-11e8-859f-eadc403c6207:2321 Feb 14 12:48:04 zabbix-db-node2 mysqld[3150]: Local state: ad8e3ac3-116d-11e8-859f-eadc403c6207:-1 Feb 14 12:48:04 zabbix-db-node2 mysqld[3150]: 2018-02-14 12:48:04 139646100187904 [Note] WSREP: New cluster view: global state: ad8e3ac3-116d-11e8-859f-eadc403c6207:2321, view# 4: Primary, number o Feb 14 12:48:04 zabbix-db-node2 mysqld[3150]: 2018-02-14 12:48:04 139646100187904 [Warning] WSREP: Gap in state sequence. Need state transfer. Feb 14 12:48:04 zabbix-db-node2 mysqld[3150]: 2018-02-14 12:48:04 139645769742080 [Note] WSREP: Running: 'wsrep\_sst\_xtrabackup --role ‘joiner’ --address ‘10.10.16.2’ --datadir ‘/var/lib/mysql/’ - Feb 14 12:48:04 zabbix-db-node2 mysqld[3150]: /usr//bin/wsrep\_sst\_xtrabackup: line 397: setup\_ports: command not found Feb 14 12:48:04 zabbix-db-node2 mysqld[3150]: 2018-02-14 12:48:04 139645769742080 [ERROR] WSREP: Failed to read ‘ready ’ from: wsrep\_sst\_xtrabackup --role ‘joiner’ --address ‘10.10.16.2’ --da Feb 14 12:48:04 zabbix-db-node2 mysqld[3150]: Read: ‘(null)’ Feb 14 12:48:04 zabbix-db-node2 mysqld[3150]: 2018-02-14 12:48:04 139645769742080 [ERROR] WSREP: Process completed with error: wsrep\_sst\_xtrabackup --role ‘joiner’ --address ‘10.10.16.2’ --datadir Feb 14 12:48:04 zabbix-db-node2 mysqld[3150]: 2018-02-14 12:48:04 139646100187904 [ERROR] WSREP: Failed to prepare for ‘xtrabackup’ SST. Unrecoverable. Feb 14 12:48:04 zabbix-db-node2 mysqld[3150]: 2018-02-14 12:48:04 139646100187904 [ERROR] Aborting Feb 14 12:48:18 zabbix-db-node2 mysqld[3150]: Error in my\_thread\_global\_end(): 1 threads didn’t exit Feb 14 12:48:18 zabbix-db-node2 systemd[1]: mariadb.service: main process exited, code=exited, status=1/FAILURE Feb 14 12:48:18 zabbix-db-node2 systemd[1]: Failed to start MariaDB 10.1.31 database server. Feb 14 12:48:18 zabbix-db-node2 systemd[1]: Unit mariadb.service entered failed state. Feb 14 12:48:18 zabbix-db-node2 systemd[1]: mariadb.service failed. What is wrong? Thank you! ---
**Author:** ![sidharrell](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/sidharrell/32/1168_2.png) [@sidharrell](https://forums.percona.com/u/sidharrell)\ **Post date:** [February 19, 2018, 4:34pm UTC](https://forums.percona.com/t/mariadb-galera-cluster-with-xtrabackup-issue/6195/2 "2018-02-19T16:34:06Z")
I’ve just hit the same problem. It appears to be a problem of interaction of the latest version 10.1.31 with innobackupex. We’ve switched some systems back to 10.1.30, which fixes it, and others by switching to rsync, which fixes it, but either solution is not long term. I’m looking for a bug report now. Will update if I find it. ---
**Author:** ![sidharrell](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/sidharrell/32/1168_2.png) [@sidharrell](https://forums.percona.com/u/sidharrell)\ **Post date:** [February 19, 2018, 4:35pm UTC](https://forums.percona.com/t/mariadb-galera-cluster-with-xtrabackup-issue/6195/3 "2018-02-19T16:35:13Z")
[url][[MDEV-15254] 10.1.31 does not join an existing cluster with SST xtrabackup-v2 - Jira](https://jira.mariadb.org/browse/MDEV-15254?attachmentViewMode=list%5B/url%5D)