Unable to bring up second node in my cluster

MySQL & MariaDB Percona XtraDB Cluster 5.x
1

I have 3 VMs configured in RHEL 6.5 … latest percona release. The first node builds fine and starts up as expected. I create the database and tablespaces on the second node … however when attempting to start I get:
2014-07-02 12:35:39 29662 [Note] WSREP: Shifting PRIMARY → JOINER (TO: 28675028)
2014-07-02 12:35:39 29662 [Note] WSREP: Requesting state transfer: success, donor: 0
2014-07-02 12:35:57 29662 [Note] WSREP: 0 (uatchaldrondb1): State transfer to 1 (uatchaldrondb3) complete.
2014-07-02 12:35:57 29662 [Note] WSREP: Member 0 (uatchaldrondb1) synced with group.
WSREP_SST: [INFO] Joiner cleanup. (20140702 12:35:58.185)
WSREP_SST: [INFO] Joiner cleanup done. (20140702 12:35:58.693)
2014-07-02 12:35:58 29662 [Note] WSREP: SST complete, seqno: 28675028
2014-07-02 12:35:58 29662 [Note] Plugin ‘FEDERATED’ is disabled.
2014-07-02 12:35:58 29662 [Note] InnoDB: The InnoDB memory heap is disabled
2014-07-02 12:35:58 29662 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2014-07-02 12:35:58 29662 [Note] InnoDB: Compressed tables use zlib 1.2.3
2014-07-02 12:35:58 29662 [Note] InnoDB: Using Linux native AIO
2014-07-02 12:35:58 29662 [Note] InnoDB: Using CPU crc32 instructions
2014-07-02 12:35:58 29662 [Note] InnoDB: Initializing buffer pool, size = 3.0G
2014-07-02 12:35:58 29662 [Note] InnoDB: Completed initialization of buffer pool
2014-07-02 12:35:58 29662 [Note] InnoDB: Highest supported file format is Barracuda.
2014-07-02 12:35:58 29662 [Note] InnoDB: Log scan progressed past the checkpoint lsn 39408212788
2014-07-02 12:35:58 29662 [Note] InnoDB: Database was not shutdown normally!
2014-07-02 12:35:58 29662 [Note] InnoDB: Starting crash recovery.
2014-07-02 12:35:58 29662 [Note] InnoDB: Reading tablespace information from the .ibd files…
2014-07-02 12:35:58 29662 [ERROR] InnoDB: Attempted to open a previously opened tablespace. Previous tablespace mysql/innodb_table_stats uses space ID: 1 at filepath: ./mysql/innodb_table_stats.ibd. Cannot open tablespace edw_target/png_edw_guid which uses space ID: 1 at filepath: /usr/local/sg/mysqldata/services/edw_target/png_edw_guid.ibd
2014-07-02 12:35:58 7f58aef617e0 InnoDB: Operating system error number 2 in a file operation.
InnoDB: The error means the system cannot find the path specified.
InnoDB: If you are installing InnoDB, remember that you must create
InnoDB: directories yourself, InnoDB does not create them.
InnoDB: Error: could not open single-table tablespace file ./edw_target/png_edw_guid.ibd
InnoDB: We do not continue the crash recovery, because the table may become
InnoDB: corrupt if we cannot apply the log records in the InnoDB log to it.
InnoDB: To fix the problem and start mysqld:
InnoDB: 1) If there is a permission problem in the file and mysqld cannot
InnoDB: open the file, you should modify the permissions.
InnoDB: 2) If the table is not needed, or you can restore it from a backup,
InnoDB: then you can remove the .ibd file, and InnoDB will do a normal
InnoDB: crash recovery and ignore that table.
InnoDB: 3) If the file system or the disk is broken, and you cannot remove
InnoDB: the .ibd file, you can set innodb_force_recovery > 0 in my.cnf
InnoDB: and force InnoDB to continue crash recovery here.
140702 12:35:58 mysqld_safe mysqld from pid file /usr/local/sg/chaldron/mysql/data/mysql/usphlliweb89.pid ended

Permissions are set to same as on the first server. It must be getting the db log files from the first node since there is no data in this node.
Any ideas?

2

Hi,

2014-07-02 12:35:39 29662 [Note] WSREP: Shifting PRIMARY → JOINER (TO: 28675028)
2014-07-02 12:35:39 29662 [Note] WSREP: Requesting state transfer: success, donor: 0
2014-07-02 12:35:57 29662 [Note] WSREP: 0 (uatchaldrondb1): State transfer to 1 (uatchaldrondb3) complete.
2014-07-02 12:35:57 29662 [Note] WSREP: Member 0 (uatchaldrondb1) synced with group.
WSREP_SST: [INFO] Joiner cleanup. (20140702 12:35:58.185)
WSREP_SST: [INFO] Joiner cleanup done. (20140702 12:35:58.693)
2014-07-02 12:35:58 29662 [Note] WSREP: SST complete, seqno: 28675028

It seems state transfer didn’t finish normally because

  • writeset position(28675028) doesn’t seems to be changed
  • no file cleanup log ( normally it seems we can see the log of the deleted files )

clear 2nd node data directory and restart might work well.

The original problem(InnoDB: Attempted to open a previously opened tablespace) might hit windows Installer related bug as follows.
[URL]MySQL Bugs: #67595: Conflicting table spaces upon application restart

3

This is the error I am getting from innodbbackup.log:
A tcp dump shows the joiner hanging up the tcp request immediately … I have turned off the firewalls for these tests

InnoDB Backup Utility v1.5.1-xtrabackup; Copyright 2003, 2009 Innobase Oy
and Percona LLC and/or its affiliates 2009-2013. All Rights Reserved.

This software is published under
the GNU GENERAL PUBLIC LICENSE Version 2, June 1991.

Get the latest version of Percona XtraBackup, documentation, and help resources:
[url]Percona XtraBackup - MySQL Database Backup Software

140908 11:56:46 innobackupex: Connecting to MySQL server with DSN ‘dbi:mysql:;mysql_read_default_file=/etc/my.cnf;mysql_read_default_group=xtrabackup;mysql_socket=/var/lib/mysql/mysql.sock’ as ‘admin’ (using password: YES).
140908 11:56:46 innobackupex: Connected to MySQL server
IMPORTANT: Please check that the backup run completes successfully.
At the end of a successful backup run innobackupex
prints “completed OK!”.

innobackupex: Using mysql server version 5.6.15-56-log

innobackupex: Created backup directory /tmp/tmp.iotsptp2sG
^Gxbstream: Error writing file ‘UNOPENED’ (Errcode: 32 - Broken pipe)
innobackupex: ‘xbstream -c’ returned with exit code 1.
innobackupex: Error: Failed to stream ‘/tmp/tmp.FmtmkMcRUl/backup-my.cnf’: 1 at /usr//bin/innobackupex line 4745.

4

“2014-07-02 12:35:58 29662 [ERROR] InnoDB: Attempted to open a previously opened tablespace. Previous tablespace mysql/innodb_table_stats uses space ID: 1 at filepath: ./mysql/innodb_table_stats.ibd. Cannot open tablespace edw_target/png_edw_guid which uses space ID: 1 at filepath: /usr/local/sg/mysqldata/services/edw_target/png_edw_guid.ibd
2014-07-02 12:35:58 7f58aef617e0 InnoDB: Operating system error number 2 in a file operation.”

Have you disabled SELinux on all nodes? Paste your my.cnf file and attach innobackup.backup.log from donor and innobackup.prepare.log from joiner.