# Unable to bring up second node in my cluster **URL:** \ **Category:** Percona XtraDB Cluster 5.x\ **Created:** [July 2, 2014, 11:54am UTC](https://forums.percona.com/t/unable-to-bring-up-second-node-in-my-cluster/3591 "2014-07-02T11:54:48Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Irvin\_Betts](https://avatars.discourse-cdn.com/v4/letter/i/ee59a6/32.png) [@Irvin\_Betts](https://forums.percona.com/u/Irvin_Betts)\ **Post date:** [July 2, 2014, 11:54am UTC](https://forums.percona.com/t/unable-to-bring-up-second-node-in-my-cluster/3591/1 "2014-07-02T11:54:48Z")
I have 3 VMs configured in RHEL 6.5 … latest percona release. The first node builds fine and starts up as expected. I create the database and tablespaces on the second node … however when attempting to start I get: 2014-07-02 12:35:39 29662 [Note] WSREP: Shifting PRIMARY → JOINER (TO: 28675028) 2014-07-02 12:35:39 29662 [Note] WSREP: Requesting state transfer: success, donor: 0 2014-07-02 12:35:57 29662 [Note] WSREP: 0 (uatchaldrondb1): State transfer to 1 (uatchaldrondb3) complete. 2014-07-02 12:35:57 29662 [Note] WSREP: Member 0 (uatchaldrondb1) synced with group. WSREP\_SST: [INFO] Joiner cleanup. (20140702 12:35:58.185) WSREP\_SST: [INFO] Joiner cleanup done. (20140702 12:35:58.693) 2014-07-02 12:35:58 29662 [Note] WSREP: SST complete, seqno: 28675028 2014-07-02 12:35:58 29662 [Note] Plugin ‘FEDERATED’ is disabled. 2014-07-02 12:35:58 29662 [Note] InnoDB: The InnoDB memory heap is disabled 2014-07-02 12:35:58 29662 [Note] InnoDB: Mutexes and rw\_locks use GCC atomic builtins 2014-07-02 12:35:58 29662 [Note] InnoDB: Compressed tables use zlib 1.2.3 2014-07-02 12:35:58 29662 [Note] InnoDB: Using Linux native AIO 2014-07-02 12:35:58 29662 [Note] InnoDB: Using CPU crc32 instructions 2014-07-02 12:35:58 29662 [Note] InnoDB: Initializing buffer pool, size = 3.0G 2014-07-02 12:35:58 29662 [Note] InnoDB: Completed initialization of buffer pool 2014-07-02 12:35:58 29662 [Note] InnoDB: Highest supported file format is Barracuda. 2014-07-02 12:35:58 29662 [Note] InnoDB: Log scan progressed past the checkpoint lsn 39408212788 2014-07-02 12:35:58 29662 [Note] InnoDB: Database was not shutdown normally! 2014-07-02 12:35:58 29662 [Note] InnoDB: Starting crash recovery. 2014-07-02 12:35:58 29662 [Note] InnoDB: Reading tablespace information from the .ibd files… 2014-07-02 12:35:58 29662 [ERROR] InnoDB: Attempted to open a previously opened tablespace. Previous tablespace mysql/innodb\_table\_stats uses space ID: 1 at filepath: ./mysql/innodb\_table\_stats.ibd. Cannot open tablespace edw\_target/png\_edw\_guid which uses space ID: 1 at filepath: /usr/local/sg/mysqldata/services/edw\_target/png\_edw\_guid.ibd 2014-07-02 12:35:58 7f58aef617e0 InnoDB: Operating system error number 2 in a file operation. InnoDB: The error means the system cannot find the path specified. InnoDB: If you are installing InnoDB, remember that you must create InnoDB: directories yourself, InnoDB does not create them. InnoDB: Error: could not open single-table tablespace file ./edw\_target/png\_edw\_guid.ibd InnoDB: We do not continue the crash recovery, because the table may become InnoDB: corrupt if we cannot apply the log records in the InnoDB log to it. InnoDB: To fix the problem and start mysqld: InnoDB: 1) If there is a permission problem in the file and mysqld cannot InnoDB: open the file, you should modify the permissions. InnoDB: 2) If the table is not needed, or you can restore it from a backup, InnoDB: then you can remove the .ibd file, and InnoDB will do a normal InnoDB: crash recovery and ignore that table. InnoDB: 3) If the file system or the disk is broken, and you cannot remove InnoDB: the .ibd file, you can set innodb\_force\_recovery \> 0 in my.cnf InnoDB: and force InnoDB to continue crash recovery here. 140702 12:35:58 mysqld\_safe mysqld from pid file /usr/local/sg/chaldron/mysql/data/mysql/usphlliweb89.pid ended Permissions are set to same as on the first server. It must be getting the db log files from the first node since there is no data in this node. Any ideas? ---
**Author:** ![taka-h](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/taka-h/32/1023_2.png) [@taka-h](https://forums.percona.com/u/taka-h)\ **Post date:** [July 23, 2014, 6:56am UTC](https://forums.percona.com/t/unable-to-bring-up-second-node-in-my-cluster/3591/2 "2014-07-23T06:56:16Z")
Hi, 2014-07-02 12:35:39 29662 [Note] WSREP: Shifting PRIMARY → JOINER (TO: 28675028) 2014-07-02 12:35:39 29662 [Note] WSREP: Requesting state transfer: success, donor: 0 2014-07-02 12:35:57 29662 [Note] WSREP: 0 (uatchaldrondb1): State transfer to 1 (uatchaldrondb3) complete. 2014-07-02 12:35:57 29662 [Note] WSREP: Member 0 (uatchaldrondb1) synced with group. WSREP\_SST: [INFO] Joiner cleanup. (20140702 12:35:58.185) WSREP\_SST: [INFO] Joiner cleanup done. (20140702 12:35:58.693) 2014-07-02 12:35:58 29662 [Note] WSREP: SST complete, seqno: 28675028 It seems state transfer didn’t finish normally because - writeset position(28675028) doesn’t seems to be changed - no file cleanup log ( normally it seems we can see the log of the deleted files ) clear 2nd node data directory and restart might work well. The original problem(InnoDB: Attempted to open a previously opened tablespace) might hit windows Installer related bug as follows. [URL][MySQL Bugs: #67595: Conflicting table spaces upon application restart](http://bugs.mysql.com/bug.php?id=67595%5B/URL%5D) ---
**Author:** ![Irvin\_Betts](https://avatars.discourse-cdn.com/v4/letter/i/ee59a6/32.png) [@Irvin\_Betts](https://forums.percona.com/u/Irvin_Betts)\ **Post date:** [September 8, 2014, 10:31am UTC](https://forums.percona.com/t/unable-to-bring-up-second-node-in-my-cluster/3591/3 "2014-09-08T10:31:16Z")
This is the error I am getting from innodbbackup.log: A tcp dump shows the joiner hanging up the tcp request immediately … I have turned off the firewalls for these tests InnoDB Backup Utility v1.5.1-xtrabackup; Copyright 2003, 2009 Innobase Oy and Percona LLC and/or its affiliates 2009-2013. All Rights Reserved. This software is published under the GNU GENERAL PUBLIC LICENSE Version 2, June 1991. Get the latest version of Percona XtraBackup, documentation, and help resources: [url][Percona XtraBackup - MySQL Database Backup Software](http://www.percona.com/xb/p%5B/url%5D) 140908 11:56:46 innobackupex: Connecting to MySQL server with DSN ‘dbi:mysql:;mysql\_read\_default\_file=/etc/my.cnf;mysql\_read\_default\_group=xtrabackup;mysql\_socket=/var/lib/mysql/mysql.sock’ as ‘admin’ (using password: YES). 140908 11:56:46 innobackupex: Connected to MySQL server IMPORTANT: Please check that the backup run completes successfully. At the end of a successful backup run innobackupex prints “completed OK!”. innobackupex: Using mysql server version 5.6.15-56-log innobackupex: Created backup directory /tmp/tmp.iotsptp2sG ^Gxbstream: Error writing file ‘UNOPENED’ (Errcode: 32 - Broken pipe) innobackupex: ‘xbstream -c’ returned with exit code 1. innobackupex: Error: Failed to stream ‘/tmp/tmp.FmtmkMcRUl/backup-my.cnf’: 1 at /usr//bin/innobackupex line 4745. ---
**Author:** ![przemek](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/przemek/32/3_2.png) [@przemek](https://forums.percona.com/u/przemek)\ **Post date:** [October 2, 2014, 12:34pm UTC](https://forums.percona.com/t/unable-to-bring-up-second-node-in-my-cluster/3591/4 "2014-10-02T12:34:46Z")
“2014-07-02 12:35:58 29662 [ERROR] InnoDB: Attempted to open a previously opened tablespace. Previous tablespace mysql/innodb\_table\_stats uses space ID: 1 at filepath: ./mysql/innodb\_table\_stats.ibd. Cannot open tablespace edw\_target/png\_edw\_guid which uses space ID: 1 at filepath: /usr/local/sg/mysqldata/services/edw\_target/png\_edw\_guid.ibd 2014-07-02 12:35:58 7f58aef617e0 InnoDB: Operating system error number 2 in a file operation.” Have you disabled SELinux on all nodes? Paste your my.cnf file and attach innobackup.backup.log from donor and innobackup.prepare.log from joiner.