Cannot join second node

MySQL & MariaDB Percona XtraDB Cluster 5.x
1

I have been trying to set up a new XtraDB Cluster like several times before, following this HOWTO: [URL=“http://www.percona.com/doc/percona-xtradb-cluster/5.5/howtos/cenots_howto.html”]http://www.percona.com/doc/percona-x...ots_howto.html[/URL].

After setting up the first node with an appropriate user for sst, I’ve now been trying to start the second node for a day. On the donor side, mysql.log and innobackup.backup.log look perfectly fine, but the joiner mysql.log tells me that xtrabackup_galera_info wasn’t created, and wsrep_sst_xtrabackup terminates with code 32 (broken pipe). I have searched all archives I could get hold of, but could not find anything appropriate. Most people are pointing to error messages in the innobackup.backup.log, but this really seems to be clean in my case. Does anybody have any idea? I would be really extremely grateful for any hint.

my.cnf.donor.txt (2.41 KB)

my.cnf.joiner.txt (2.41 KB)

innobackup.backup.log.donor.txt (4.09 KB)

mysqld.log.joiner.txt (14.8 KB)

mysqld.log.donor.txt (5.75 KB)

2

I really can’t convince innobackupex to create galera_info. The call is

Where should I expect galera_info? In /tmp maybe or /var/lib/mysql? Nothing… :frowning:

[DRUPAL-LIVE] root@drup1 sudo -u mysql innobackupex --defaults-file=/etc/my.cnf --user=sstuser --password=UmRgRe77sjOmshuXm8odtt --socket=/var/lib/mysql/mysql.sock --galera-info --stream=tar /tmp >/tmp/innobackupex.out

InnoDB Backup Utility v1.5.1-xtrabackup; Copyright 2003, 2009 Innobase Oy
and Percona LLC and/or its affiliates 2009-2013. All Rights Reserved.

This software is published under
the GNU GENERAL PUBLIC LICENSE Version 2, June 1991.

Get the latest version of Percona XtraBackup, documentation, and help resources:
http://www.percona.com/xb/p

141210 14:29:35 innobackupex: Connecting to MySQL server with DSN ‘dbi:mysql:;mysql_read_default_file=/etc/my.cnf;mysql_read_default_group=xtrabackup;mysql_socket=/var/lib/mysql/mysql.sock’ as ‘sstuser’ (using password: YES).
141210 14:29:35 innobackupex: Connected to MySQL server
141210 14:29:35 innobackupex: Executing a version check against the server…
141210 14:29:35 innobackupex: Done.
141210 14:29:35 innobackupex: Starting the backup operation

IMPORTANT: Please check that the backup run completes successfully.
At the end of a successful backup run innobackupex
prints “completed OK!”.

innobackupex: Using server version 5.6.21-70.1-56-log

innobackupex: Created backup directory /tmp

141210 14:29:35 innobackupex: Starting ibbackup with command: xtrabackup --defaults-file=“/etc/my.cnf” --defaults-group=“mysqld” --backup --suspend-at-end --target-dir=/tmp --tmpdir=/tmp --extra-lsndir=‘/tmp’ --stream=tar
innobackupex: Waiting for ibbackup (pid=15401) to suspend
innobackupex: Suspend file ‘/tmp/xtrabackup_suspended_2’

xtrabackup version 2.2.6 based on MySQL server 5.6.21 Linux (x86_64) (revision id: )
xtrabackup: uses posix_fadvise().
xtrabackup: cd to /var/lib/mysql
xtrabackup: open files limit requested 0, set to 1024000
xtrabackup: using the following InnoDB configuration:
xtrabackup: innodb_data_home_dir = ./
xtrabackup: innodb_data_file_path = ibdata1:10M:autoextend
xtrabackup: innodb_log_group_home_dir = ./
xtrabackup: innodb_log_files_in_group = 3
xtrabackup: innodb_log_file_size = 268435456

log scanned up to (1626850)
xtrabackup: Generating a list of tablespaces
[01] Streaming ./ibdata1
[01] …done
[01] Streaming ./mysql/slave_worker_info.ibd
[01] …done
[01] Streaming ./mysql/innodb_table_stats.ibd
[01] …done
[01] Streaming ./mysql/innodb_index_stats.ibd
[01] …done
[01] Streaming ./mysql/slave_relay_log_info.ibd
[01] …done
[01] Streaming ./mysql/slave_master_info.ibd
[01] …done
log scanned up to (1626850)
xtrabackup: Creating suspend file ‘/tmp/xtrabackup_suspended_2’ with pid ‘15401’

141210 14:29:37 innobackupex: Continuing after ibbackup has suspended
141210 14:29:37 innobackupex: Executing LOCK TABLES FOR BACKUP…
141210 14:29:37 innobackupex: Backup tables lock acquired

141210 14:29:37 innobackupex: Starting to backup non-InnoDB tables and files
innobackupex: in subdirectories of ‘/var/lib/mysql/’
innobackupex: Backing up files ‘/var/lib/mysql//performance_schema/*.{frm,isl,MYD,MYI,MAD,MAI,MRG,TRG,TRN,ARM,ARZ,CSM,CSV,opt,par}’ (53 files)

log scanned up to (1626850)
innobackupex: Backing up files ‘/var/lib/mysql//mysql/*.{frm,isl,MYD,MYI,MAD,MAI,MRG,TRG,TRN,ARM,ARZ,CSM,CSV,opt,par}’ (74 files)
141210 14:29:38 innobackupex: Finished backing up non-InnoDB tables and files

141210 14:29:38 innobackupex: Executing LOCK BINLOG FOR BACKUP…
141210 14:29:38 innobackupex: Executing FLUSH ENGINE LOGS…
141210 14:29:38 innobackupex: Waiting for log copying to finish

xtrabackup: The latest check point (for incremental): ‘1626850’
xtrabackup: Stopping log copying thread.
.>> log scanned up to (1626850)

xtrabackup: Creating suspend file ‘/tmp/xtrabackup_log_copied’ with pid ‘15401’
xtrabackup: Transaction log of lsn (1626850) to (1626850) was copied.
141210 14:29:39 innobackupex: Executing UNLOCK BINLOG
141210 14:29:39 innobackupex: Executing UNLOCK TABLES
141210 14:29:39 innobackupex: All tables unlocked

innobackupex: Backup created in directory ‘/tmp’
innobackupex: MySQL binlog position: filename ‘mysql-bin.000006’, position 875, GTID of the last change ‘’
141210 14:29:39 innobackupex: Connection to database server closed
innobackupex: You must use -i (–ignore-zeros) option for extraction of the tar stream.
141210 14:29:39 innobackupex: completed OK!

3

Actually this seems to be the same issue as [url]http://www.percona.com/forums/questions-discussions/percona-xtradb-cluster/27626-2nd-node-unable-to-join-cluster[/url]

4

On the joiner, the following files are created:

-rw-rw---- 1 mysql mysql 23 Dec 10 16:53 xtrabackup_binlog_info
-rw-rw---- 1 root root 89 Dec 10 16:53 xtrabackup_checkpoints
-rw-rw---- 1 mysql mysql 675 Dec 10 16:53 xtrabackup_info
-rw-rw---- 1 root root 2.5K Dec 10 16:53 xtrabackup_logfile

In my working installations of XtraDB Cluster, there’s also xtrabackup_galera_info, the one that’s apparently not being created here.

Strangely, some of the files belong to root, others to the mysql user.

5

I reinstalled PXC from scratch, removed the old datadirs and used wsrep_sst_method=xtrabackup-v2 this time. I followed [url]Percona XtraDB Cluster step by step.

The joiner does not come up, and the log is full with messages like

^G/usr/sbin/mysqld: Can’t find file: ‘./mysql/plugin.frm’ (errno: 13 - Permission denied)
2014-12-11 02:10:41 29779 [ERROR] Can’t open the mysql.plugin table. Please run mysql_upgrade to create it.
2014-12-11 02:10:41 29779 [Note] InnoDB: Using atomics to ref count buffer pool pages
2014-12-11 02:10:41 29779 [Note] InnoDB: The InnoDB memory heap is disabled
2014-12-11 02:10:41 29779 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2014-12-11 02:10:41 29779 [Note] InnoDB: Memory barrier is not used
2014-12-11 02:10:41 29779 [Note] InnoDB: Compressed tables use zlib 1.2.3
2014-12-11 02:10:41 29779 [Note] InnoDB: Using Linux native AIO
2014-12-11 02:10:41 29779 [Note] InnoDB: Using CPU crc32 instructions
2014-12-11 02:10:41 29779 [Note] InnoDB: Initializing buffer pool, size = 4.0G
2014-12-11 02:10:41 29779 [Note] InnoDB: Completed initialization of buffer pool
2014-12-11 02:10:41 29779 [Note] InnoDB: Highest supported file format is Barracuda.
2014-12-11 02:10:41 7f6a5f35f800 InnoDB: Operating system error number 13 in a file operation.
InnoDB: The error means mysqld does not have the access rights to
InnoDB: the directory.
2014-12-11 02:10:41 29779 [ERROR] InnoDB: Could not find a valid tablespace file for ‘mysql/innodb_index_stats’. See http://dev.mysql.com/doc/refman/5.6/…-datadict.html for how to resolve the issue.
2014-12-11 02:10:41 29779 [ERROR] InnoDB: Tablespace open failed for ‘“mysql”.“innodb_index_stats”’, ignored.
2014-12-11 02:10:41 7f6a5f35f800 InnoDB: Operating system error number 13 in a file operation.
InnoDB: The error means mysqld does not have the access rights to
InnoDB: the directory.
2014-12-11 02:10:41 29779 [ERROR] InnoDB: Could not find a valid tablespace file for ‘mysql/innodb_table_stats’. See http://dev.mysql.com/doc/refman/5.6/…-datadict.html for how to resolve the issue.

Some of the content in the datadir on the joiner does in fact belong to root:
[LIVE] root@drup1 ls -l /var/lib/mysql/
total 973M
-rw-rw---- 1 mysql mysql 56 Dec 11 02:10 auto.cnf
-rw-rw---- 1 root root 359 Dec 11 02:10 backup-my.cnf
-rw------- 1 mysql mysql 129M Dec 11 02:10 galera.cache
-rw-rw---- 1 mysql mysql 104 Dec 11 02:10 grastate.dat
-rw-rw---- 1 mysql mysql 217 Dec 11 02:10 gvwstate.dat
-rw-rw---- 1 root root 962 Dec 11 02:10 ib_buffer_pool
-rw-rw---- 1 root root 74M Dec 11 02:10 ibdata1
-rw-rw---- 1 root root 256M Dec 11 02:10 ib_logfile0
-rw-rw---- 1 root root 256M Dec 11 02:10 ib_logfile1
-rw-rw---- 1 root root 256M Dec 11 02:10 ib_logfile2
-rw-rw---- 1 root root 5.5K Dec 11 02:10 innobackup.prepare.log
drwx------ 2 root root 4.0K Dec 11 02:10 mysql
-rw-rw---- 1 mysql mysql 120 Dec 11 02:10 mysql-bin.000001
-rw-rw---- 1 mysql mysql 19 Dec 11 02:10 mysql-bin.index
drwx------ 2 root root 4.0K Dec 11 02:10 performance_schema
-rw-r–r-- 1 root root 118 Dec 11 02:07 RPM_UPGRADE_HISTORY
drwx------ 2 root root 4.0K Dec 11 02:10 test
-rw-rw---- 1 root root 21 Dec 11 02:10 xtrabackup_binlog_info
-rw-rw---- 1 root root 89 Dec 11 02:10 xtrabackup_checkpoints
-rw-rw---- 1 root root 38 Dec 11 02:10 xtrabackup_galera_info
-rw-rw---- 1 root root 720 Dec 11 02:10 xtrabackup_info
-rw-rw---- 1 root root 2.0M Dec 11 02:10 xtrabackup_logfile

This is a new installation, and the user in my.cnf is mysql, not root. I do start the init script as root, but I think this is normal.

6

How did you prepare the joiner before starting with wsrep-enabled my.cnf? Is the /var/lib/mysql/ owned by mysql user and group?
Also if you do restore backup manually from donor, remember to chown all files to mysql user/group too.
Another thing is when you initialize datadir with mysql_install_db, use the “–user=mysql” option.

7

Thank you for your reply. I got the cluster to work with an older version. Now it also works again with the latest version available on the yum repo. Apparently it was an issue with a build that was out for a couple of days / weeks.