# Cannot join second node **URL:** \ **Category:** Percona XtraDB Cluster 5.x\ **Created:** [December 10, 2014, 3:44am UTC](https://forums.percona.com/t/cannot-join-second-node/3938 "2014-12-10T03:44:56Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![SouthernBelle](https://avatars.discourse-cdn.com/v4/letter/s/90ced4/32.png) [@SouthernBelle](https://forums.percona.com/u/SouthernBelle)\ **Post date:** [December 10, 2014, 3:44am UTC](https://forums.percona.com/t/cannot-join-second-node/3938/1 "2014-12-10T03:44:56Z")
I have been trying to set up a new XtraDB Cluster like several times before, following this HOWTO: [URL=“[http://www.percona.com/doc/percona-xtradb-cluster/5.5/howtos/cenots\_howto.html](http://www.percona.com/doc/percona-xtradb-cluster/5.5/howtos/cenots_howto.html)”][http://www.percona.com/doc/percona-x...ots\_howto.html[/URL]](http://www.percona.com/doc/percona-x...ots_howto.html%5B/URL%5D). After setting up the first node with an appropriate user for sst, I’ve now been trying to start the second node for a day. On the donor side, mysql.log and innobackup.backup.log look perfectly fine, but the joiner mysql.log tells me that xtrabackup\_galera\_info wasn’t created, and wsrep\_sst\_xtrabackup terminates with code 32 (broken pipe). I have searched all archives I could get hold of, but could not find anything appropriate. Most people are pointing to error messages in the innobackup.backup.log, but this really seems to be clean in my case. Does anybody have any idea? I would be really extremely grateful for any hint. [my.cnf.donor.txt](https://forums.percona.com/uploads/short-url/gI2d2C90xcoH1TQrcom7U26bhTG.txt) (2.41 KB) [my.cnf.joiner.txt](https://forums.percona.com/uploads/short-url/8Y9zDaAeIPjk661qvyps4uHHMSh.txt) (2.41 KB) [innobackup.backup.log.donor.txt](https://forums.percona.com/uploads/short-url/ykt7qxbveAF3RT36X7z8CK29dBg.txt) (4.09 KB) [mysqld.log.joiner.txt](https://forums.percona.com/uploads/short-url/eG9j351udEFjdrhxT23beTpoEea.txt) (14.8 KB) [mysqld.log.donor.txt](https://forums.percona.com/uploads/short-url/mwQIq37VqszbHUoMC4fwjYLDWJ2.txt) (5.75 KB) ---
**Author:** ![SouthernBelle](https://avatars.discourse-cdn.com/v4/letter/s/90ced4/32.png) [@SouthernBelle](https://forums.percona.com/u/SouthernBelle)\ **Post date:** [December 10, 2014, 8:01am UTC](https://forums.percona.com/t/cannot-join-second-node/3938/2 "2014-12-10T08:01:49Z")
I really can’t convince innobackupex to create galera\_info. The call is Where should I expect galera\_info? In /tmp maybe or /var/lib/mysql? Nothing… ☹ [DRUPAL-LIVE] root@drup1 sudo -u mysql innobackupex --defaults-file=/etc/my.cnf --user=sstuser --password=UmRgRe77sjOmshuXm8odtt --socket=/var/lib/mysql/mysql.sock --galera-info --stream=tar /tmp \>/tmp/innobackupex.out InnoDB Backup Utility v1.5.1-xtrabackup; Copyright 2003, 2009 Innobase Oy and Percona LLC and/or its affiliates 2009-2013. All Rights Reserved. This software is published under the GNU GENERAL PUBLIC LICENSE Version 2, June 1991. Get the latest version of Percona XtraBackup, documentation, and help resources: [http://www.percona.com/xb/p](http://www.percona.com/xb/p) 141210 14:29:35 innobackupex: Connecting to MySQL server with DSN ‘dbi:mysql:;mysql\_read\_default\_file=/etc/my.cnf;mysql\_read\_default\_group=xtrabackup;mysql\_socket=/var/lib/mysql/mysql.sock’ as ‘sstuser’ (using password: YES). 141210 14:29:35 innobackupex: Connected to MySQL server 141210 14:29:35 innobackupex: Executing a version check against the server… 141210 14:29:35 innobackupex: Done. 141210 14:29:35 innobackupex: Starting the backup operation IMPORTANT: Please check that the backup run completes successfully. At the end of a successful backup run innobackupex prints “completed OK!”. innobackupex: Using server version 5.6.21-70.1-56-log innobackupex: Created backup directory /tmp 141210 14:29:35 innobackupex: Starting ibbackup with command: xtrabackup --defaults-file=“/etc/my.cnf” --defaults-group=“mysqld” --backup --suspend-at-end --target-dir=/tmp --tmpdir=/tmp --extra-lsndir=‘/tmp’ --stream=tar innobackupex: Waiting for ibbackup (pid=15401) to suspend innobackupex: Suspend file ‘/tmp/xtrabackup\_suspended\_2’ xtrabackup version 2.2.6 based on MySQL server 5.6.21 Linux (x86\_64) (revision id: ) xtrabackup: uses posix\_fadvise(). xtrabackup: cd to /var/lib/mysql xtrabackup: open files limit requested 0, set to 1024000 xtrabackup: using the following InnoDB configuration: xtrabackup: innodb\_data\_home\_dir = ./ xtrabackup: innodb\_data\_file\_path = ibdata1:10M:autoextend xtrabackup: innodb\_log\_group\_home\_dir = ./ xtrabackup: innodb\_log\_files\_in\_group = 3 xtrabackup: innodb\_log\_file\_size = 268435456 > > log scanned up to (1626850) > > xtrabackup: Generating a list of tablespaces > > [01] Streaming ./ibdata1 > > [01] …done > > [01] Streaming ./mysql/slave\_worker\_info.ibd > > [01] …done > > [01] Streaming ./mysql/innodb\_table\_stats.ibd > > [01] …done > > [01] Streaming ./mysql/innodb\_index\_stats.ibd > > [01] …done > > [01] Streaming ./mysql/slave\_relay\_log\_info.ibd > > [01] …done > > [01] Streaming ./mysql/slave\_master\_info.ibd > > [01] …done > > log scanned up to (1626850) > > xtrabackup: Creating suspend file ‘/tmp/xtrabackup\_suspended\_2’ with pid ‘15401’ 141210 14:29:37 innobackupex: Continuing after ibbackup has suspended 141210 14:29:37 innobackupex: Executing LOCK TABLES FOR BACKUP… 141210 14:29:37 innobackupex: Backup tables lock acquired 141210 14:29:37 innobackupex: Starting to backup non-InnoDB tables and files innobackupex: in subdirectories of ‘/var/lib/mysql/’ innobackupex: Backing up files ‘/var/lib/mysql//performance\_schema/\*.{frm,isl,MYD,MYI,MAD,MAI,MRG,TRG,TRN,ARM,ARZ,CSM,CSV,opt,par}’ (53 files) > > log scanned up to (1626850) > > innobackupex: Backing up files ‘/var/lib/mysql//mysql/\*.{frm,isl,MYD,MYI,MAD,MAI,MRG,TRG,TRN,ARM,ARZ,CSM,CSV,opt,par}’ (74 files) > > 141210 14:29:38 innobackupex: Finished backing up non-InnoDB tables and files 141210 14:29:38 innobackupex: Executing LOCK BINLOG FOR BACKUP… 141210 14:29:38 innobackupex: Executing FLUSH ENGINE LOGS… 141210 14:29:38 innobackupex: Waiting for log copying to finish xtrabackup: The latest check point (for incremental): ‘1626850’ xtrabackup: Stopping log copying thread. .\>\> log scanned up to (1626850) xtrabackup: Creating suspend file ‘/tmp/xtrabackup\_log\_copied’ with pid ‘15401’ xtrabackup: Transaction log of lsn (1626850) to (1626850) was copied. 141210 14:29:39 innobackupex: Executing UNLOCK BINLOG 141210 14:29:39 innobackupex: Executing UNLOCK TABLES 141210 14:29:39 innobackupex: All tables unlocked innobackupex: Backup created in directory ‘/tmp’ innobackupex: MySQL binlog position: filename ‘mysql-bin.000006’, position 875, GTID of the last change ‘’ 141210 14:29:39 innobackupex: Connection to database server closed innobackupex: You must use -i (–ignore-zeros) option for extraction of the tar stream. 141210 14:29:39 innobackupex: completed OK! ---
**Author:** ![SouthernBelle](https://avatars.discourse-cdn.com/v4/letter/s/90ced4/32.png) [@SouthernBelle](https://forums.percona.com/u/SouthernBelle)\ **Post date:** [December 10, 2014, 8:06am UTC](https://forums.percona.com/t/cannot-join-second-node/3938/3 "2014-12-10T08:06:48Z")
Actually this seems to be the same issue as [url][http://www.percona.com/forums/questions-discussions/percona-xtradb-cluster/27626-2nd-node-unable-to-join-cluster[/url]](http://www.percona.com/forums/questions-discussions/percona-xtradb-cluster/27626-2nd-node-unable-to-join-cluster%5B/url%5D) ---
**Author:** ![SouthernBelle](https://avatars.discourse-cdn.com/v4/letter/s/90ced4/32.png) [@SouthernBelle](https://forums.percona.com/u/SouthernBelle)\ **Post date:** [December 10, 2014, 10:32am UTC](https://forums.percona.com/t/cannot-join-second-node/3938/4 "2014-12-10T10:32:17Z")
On the joiner, the following files are created: -rw-rw---- 1 mysql mysql 23 Dec 10 16:53 xtrabackup\_binlog\_info -rw-rw---- 1 root root 89 Dec 10 16:53 xtrabackup\_checkpoints -rw-rw---- 1 mysql mysql 675 Dec 10 16:53 xtrabackup\_info -rw-rw---- 1 root root 2.5K Dec 10 16:53 xtrabackup\_logfile In my working installations of XtraDB Cluster, there’s also xtrabackup\_galera\_info, the one that’s apparently not being created here. Strangely, some of the files belong to root, others to the mysql user. ---
**Author:** ![SouthernBelle](https://avatars.discourse-cdn.com/v4/letter/s/90ced4/32.png) [@SouthernBelle](https://forums.percona.com/u/SouthernBelle)\ **Post date:** [December 10, 2014, 7:51pm UTC](https://forums.percona.com/t/cannot-join-second-node/3938/5 "2014-12-10T19:51:06Z")
I reinstalled PXC from scratch, removed the old datadirs and used wsrep\_sst\_method=xtrabackup-v2 this time. I followed [url][Percona XtraDB Cluster](http://www.percona.com/doc/percona-xtradb-cluster/5.6/howtos/ubuntu_howto.html%5B/url%5D) step by step. The joiner does not come up, and the log is full with messages like ^G/usr/sbin/mysqld: Can’t find file: ‘./mysql/plugin.frm’ (errno: 13 - Permission denied) 2014-12-11 02:10:41 29779 [ERROR] Can’t open the mysql.plugin table. Please run mysql\_upgrade to create it. 2014-12-11 02:10:41 29779 [Note] InnoDB: Using atomics to ref count buffer pool pages 2014-12-11 02:10:41 29779 [Note] InnoDB: The InnoDB memory heap is disabled 2014-12-11 02:10:41 29779 [Note] InnoDB: Mutexes and rw\_locks use GCC atomic builtins 2014-12-11 02:10:41 29779 [Note] InnoDB: Memory barrier is not used 2014-12-11 02:10:41 29779 [Note] InnoDB: Compressed tables use zlib 1.2.3 2014-12-11 02:10:41 29779 [Note] InnoDB: Using Linux native AIO 2014-12-11 02:10:41 29779 [Note] InnoDB: Using CPU crc32 instructions 2014-12-11 02:10:41 29779 [Note] InnoDB: Initializing buffer pool, size = 4.0G 2014-12-11 02:10:41 29779 [Note] InnoDB: Completed initialization of buffer pool 2014-12-11 02:10:41 29779 [Note] InnoDB: Highest supported file format is Barracuda. 2014-12-11 02:10:41 7f6a5f35f800 InnoDB: Operating system error number 13 in a file operation. InnoDB: The error means mysqld does not have the access rights to InnoDB: the directory. 2014-12-11 02:10:41 29779 [ERROR] InnoDB: Could not find a valid tablespace file for ‘mysql/innodb\_index\_stats’. See [http://dev.mysql.com/doc/refman/5.6/](http://dev.mysql.com/doc/refman/5.6/)…-datadict.html for how to resolve the issue. 2014-12-11 02:10:41 29779 [ERROR] InnoDB: Tablespace open failed for ‘“mysql”.“innodb\_index\_stats”’, ignored. 2014-12-11 02:10:41 7f6a5f35f800 InnoDB: Operating system error number 13 in a file operation. InnoDB: The error means mysqld does not have the access rights to InnoDB: the directory. 2014-12-11 02:10:41 29779 [ERROR] InnoDB: Could not find a valid tablespace file for ‘mysql/innodb\_table\_stats’. See [http://dev.mysql.com/doc/refman/5.6/](http://dev.mysql.com/doc/refman/5.6/)…-datadict.html for how to resolve the issue. Some of the content in the datadir on the joiner does in fact belong to root: [LIVE] root@drup1 ls -l /var/lib/mysql/ total 973M -rw-rw---- 1 mysql mysql 56 Dec 11 02:10 auto.cnf -rw-rw---- 1 root root 359 Dec 11 02:10 backup-my.cnf -rw------- 1 mysql mysql 129M Dec 11 02:10 galera.cache -rw-rw---- 1 mysql mysql 104 Dec 11 02:10 grastate.dat -rw-rw---- 1 mysql mysql 217 Dec 11 02:10 gvwstate.dat -rw-rw---- 1 root root 962 Dec 11 02:10 ib\_buffer\_pool -rw-rw---- 1 root root 74M Dec 11 02:10 ibdata1 -rw-rw---- 1 root root 256M Dec 11 02:10 ib\_logfile0 -rw-rw---- 1 root root 256M Dec 11 02:10 ib\_logfile1 -rw-rw---- 1 root root 256M Dec 11 02:10 ib\_logfile2 -rw-rw---- 1 root root 5.5K Dec 11 02:10 innobackup.prepare.log drwx------ 2 root root 4.0K Dec 11 02:10 mysql -rw-rw---- 1 mysql mysql 120 Dec 11 02:10 mysql-bin.000001 -rw-rw---- 1 mysql mysql 19 Dec 11 02:10 mysql-bin.index drwx------ 2 root root 4.0K Dec 11 02:10 performance\_schema -rw-r–r-- 1 root root 118 Dec 11 02:07 RPM\_UPGRADE\_HISTORY drwx------ 2 root root 4.0K Dec 11 02:10 test -rw-rw---- 1 root root 21 Dec 11 02:10 xtrabackup\_binlog\_info -rw-rw---- 1 root root 89 Dec 11 02:10 xtrabackup\_checkpoints -rw-rw---- 1 root root 38 Dec 11 02:10 xtrabackup\_galera\_info -rw-rw---- 1 root root 720 Dec 11 02:10 xtrabackup\_info -rw-rw---- 1 root root 2.0M Dec 11 02:10 xtrabackup\_logfile This is a new installation, and the user in my.cnf is mysql, not root. I do start the init script as root, but I think this is normal. ---
**Author:** ![przemek](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/przemek/32/3_2.png) [@przemek](https://forums.percona.com/u/przemek)\ **Post date:** [December 23, 2014, 5:06am UTC](https://forums.percona.com/t/cannot-join-second-node/3938/6 "2014-12-23T05:06:28Z")
How did you prepare the joiner before starting with wsrep-enabled my.cnf? Is the /var/lib/mysql/ owned by mysql user and group? Also if you do restore backup manually from donor, remember to chown all files to mysql user/group too. Another thing is when you initialize datadir with mysql\_install\_db, use the “–user=mysql” option. ---
**Author:** ![SouthernBelle](https://avatars.discourse-cdn.com/v4/letter/s/90ced4/32.png) [@SouthernBelle](https://forums.percona.com/u/SouthernBelle)\ **Post date:** [January 7, 2015, 4:21pm UTC](https://forums.percona.com/t/cannot-join-second-node/3938/7 "2015-01-07T16:21:17Z")
Thank you for your reply. I got the cluster to work with an older version. Now it also works again with the latest version available on the yum repo. Apparently it was an issue with a build that was out for a couple of days / weeks.