Can't start node2 and node3 mySQL server

Hello,

I am trying to set up Percona cluster on CentOS6.5 and i followed the guide at:
[URL=“http://www.percona.com/doc/percona-xtradb-cluster/5.6/howtos/cenots_howto.html”]http://www.percona.com/doc/percona-x...ots_howto.html[/URL]

I set up the first node and bootstrapped it. Everything seems correct with that node.
The problem is with second and third node. I just can’t start mySQL server.
I get this error when using this command “/etc/init.d/mysql start”

MySQL (Percona XtraDB Cluster) is not running, but lock file (/var/lock/subsys/mysql) exists [FAILED]
Starting MySQL (Percona XtraDB Cluster).The server quit without updating PID file (/usr/local/mysql/var/localhost.localdomain.pid). [FAILED]
MySQL (Percona XtraDB Cluster) server startup failed! [FAILED]

Node 2 my.cnf:

[mysqld]
datadir=/var/lib/mysql
user=mysql

Path to Galera library

wsrep_provider=/usr/lib64/libgalera_smm.so

Cluster connection URL contains IPs of node#1, node#2 and node#3 wsrep_cluster_address=gcomm://192.168.70.123,192.168.70.120,192.168.70.144

In order for Galera to work correctly binlog format should be ROW

binlog_format=ROW

MyISAM storage engine has only experimental support

default_storage_engine=InnoDB

This changes how InnoDB autoincrement locks are managed and is a requirement for Galera innodb_autoinc_lock_mode=2

Node #2 address

wsrep_node_address=192.168.70.120

Cluster name

wsrep_cluster_name=my_centos_cluster

SST method

wsrep_sst_method=xtrabackup-v2

#Authentication for SST method
wsrep_sst_auth=“sstuser:s3cret”

Have you tried to give it a stop before starting up again? Did you create the user sstuser with password as s3cret on the first node? The configuration seems to be ok, but, try to stop it before starting it up again. If it does not work, share the error log contents of node01 and the contents of innobackup.backup.log form node02 and node03.

Hello,

I am having thesame issue with percona-xtradb-cluster with three nodes on different AWS Availabilty zones. After creating my.cnf on node2 and node3, mysql refused to start with the following error:
root@ip-10-200-12-78:~# /etc/init.d/mysql start

  • Starting MySQL (Percona XtraDB Cluster) database server mysqld
  • The server quit without updating PID file (/var/lib/mysql/ip-10-200-12-78.pid). [fail]

my.cnf configuration:

[mysqld]
datadir = /var/lib/mysql

binlog_format = ROW
innodb_buffer_pool_size=2600M
max_allowed_packet=820M
wsrep_cluster_name = pkclus
wsrep_cluster_address = gcomm://10.200.12.35,10.200.12.78,10.200.12.183
wsrep_node_address = 10.200.12.78
wsrep_provider = /usr/lib/libgalera_smm.so
wsrep_sst_method = rsync
innodb_locks_unsafe_for_binlog = 1
innodb_autoinc_lock_mode = 2
default_storage_engine=InnoDB

Note: The configuration is done accordingly on each node.

Bootstraping on the node is successful

root@ip-10-200-12-35:/var/lib/mysql# sudo service mysql restart-bootstrap

  • Stopping MySQL (Percona XtraDB Cluster) mysqld [ OK ]
  • Bootstrapping Percona XtraDB Cluster database server mysqld [ OK ]

Each node is on different AWS VPC network subnet.

Anyone to help this issue?

Thanks

This is the error log in (.err) file:

150425 14:00:29 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
150425 14:00:29 mysqld_safe WSREP: Running position recovery with --log_error=‘/var/lib/mysql/wsrep_recovery.0Ke4Rh’ --pid-file=‘/var/lib/mysql/ip-10-200-12-78-recover.pid’
2015-04-25 14:00:30 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
150425 14:00:31 mysqld_safe WSREP: Failed to recover position: 2015-04-25 14:00:30 5821 [Note] Plugin ‘FEDERATED’ is disabled. 2015-04-25 14:00:30 7f896e9f5780 InnoDB: Warning: Using innodb_locks_unsafe_for_binlog is DEPRECATED. This option may be removed in future releases. Please use READ COMMITTED transaction isolation level instead, see [url]http://dev.mysql.com/doc/refman/5.6/en/set-transaction.html[/url]. 2015-04-25 14:00:30 5821 [Note] InnoDB: The InnoDB memory heap is disabled 2015-04-25 14:00:30 5821 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2015-04-25 14:00:30 5821 [Note] InnoDB: Compressed tables use zlib 1.2.7 2015-04-25 14:00:30 5821 [Note] InnoDB: Using Linux native AIO 2015-04-25 14:00:30 5821 [Note] InnoDB: Using CPU crc32 instructions 2015-04-25 14:00:30 5821 [Note] InnoDB: Initializing buffer pool, size = 2.5G 2015-04-25 14:00:31 5821 [Note] InnoDB: Completed initialization of buffer pool 2015-04-25 14:00:31 5821 [ERROR] InnoDB: space header page consists of zero bytes in data file ./ibdata1 2015-04-25 14:00:31 5821 [ERROR] InnoDB: Could not open or create the system tablespace. If you tried to add new data files to the system tablespace, and it failed here, you should now edit innodb_data_file_path in my.cnf back to what it was, and remove the new ibdata files InnoDB created in this failed attempt. InnoDB only wrote those files full of zeros, but did not yet use them in any way. But be careful: do not remove old data files which contain your precious data! 2015-04-25 14:00:31 5821 [ERROR] Plugin ‘InnoDB’ init function returned error. 2015-04-25 14:00:31 5821 [ERROR] Plugin ‘InnoDB’ registration as a STORAGE ENGINE failed. 2015-04-25 14:00:31 5821 [ERROR] /usr/sbin/mysqld: unknown option ‘–~’ 2015-04-25 14:00:31 5821 [ERROR] Aborting 2015-04-25 14:00:31 5821 [Note] Binlog end 2015-04-25 14:00:31 5821 [Note] Shutting down plugin ‘partition’ 2015-04-25 14:00:31 5821 [Note] Shutting down plugin ‘ARCHIVE’ 2015-04-25 14:00:31 5821 [Note] Shutting down plugin ‘INNODB_CHANGED_PAGES’ 2015-04-25 14:00:31 5821 [Note] Shutting down plugin ‘INNODB_SYS_DATAFILES’ 2015-04-25 14:00:31 5821 [Note] Shutting down plugin ‘INNODB_SYS_TABLESPACES’ 2015-04-25 14:00:31 5821 [Note] Shutting down plugin ‘INNODB_SYS_FOREIGN_COLS’ 2015-04-25 14:00:31 5821 [Note] Shutting down plugin ‘INNODB_SYS_FOREIGN’ 2015-04-25 14:00:31 5821 [Note] Shutting down plugin ‘INNODB_SYS_FIELDS’ 2015-04-25 14:00:31 5821 [Note] Shutting down plugin ‘INNODB_SYS_COLUMNS’ 2015-04-25 14:00:31 5821 [Note] Shutting down plugin ‘INNODB_SYS_INDEXES’ 2015-04-25 14:00:31 5821 [Note] Shutting down plugin ‘INNODB_SYS_TABLESTATS’ 2015-04-25 14:00:31 5821 [Note] Shutting down plugin ‘INNODB_SYS_TABLES’ 2015-04-25 14:00:31 5821 [Note] Shutting down plugin ‘INNODB_FT_INDEX_TABLE’ 2015-04-25 14:00:31 5821 [Note] Shutting down plugin ‘INNODB_FT_INDEX_CACHE’ 2015-04-25 14:00:31 5821 [Note] Shutting down plugin ‘INNODB_FT_CONFIG’ 2015-04-25 14:00:31 5821 [Note] Shutting down plugin ‘INNODB_FT_BEING_DELETED’ 2015-04-25 14:00:31 5821 [Note] Shutting down plugin ‘INNODB_FT_DELETED’ 2015-04-25 14:00:31 5821 [Note] Shutting down plugin ‘INNODB_FT_DEFAULT_STOPWORD’ 2015-04-25 14:00:31 5821 [Note] Shutting down plugin ‘INNODB_METRICS’ 2015-04-25 14:00:31 5821 [Note] Shutting down plugin ‘INNODB_BUFFER_POOL_STATS’ 2015-04-25 14:00:31 5821 [Note] Shutting down plugin ‘INNODB_BUFFER_PAGE_LRU’ 2015-04-25 14:00:31 5821 [Note] Shutting down plugin ‘INNODB_BUFFER_PAGE’ 2015-04-25 14:00:31 5821 [Note] Shutting down plugin ‘INNODB_CMP_PER_INDEX_RESET’ 2015-04-25 14:00:31 5821 [Note] Shutting down plugin ‘INNODB_CMP_PER_INDEX’ 2015-04-25 14:00:31 5821 [Note] Shutting down plugin ‘INNODB_CMPMEM_RESET’ 2015-04-25 14:00:31 5821 [Note] Shutting down plugin ‘INNODB_CMPMEM’ 2015-04-25 14:00:31 5821 [Note] Shutting down plugin ‘INNODB_CMP_RESET’ 2015-04-25 14:00:31 5821 [Note] Shutting down plugin ‘INNODB_CMP’ 2015-04-25 14:00:31 5821 [Note] Shutting down plugin ‘INNODB_LOCK_WAITS’ 2015-04-25 14:00:31 5821 [Note] Shutting down plugin ‘INNODB_LOCKS’ 2015-04-25 14:00:31 5821 [Note] Shutting down plugin ‘INNODB_TRX’ 2015-04-25 14:00:31 5821 [Note] Shutting down plugin ‘XTRADB_RSEG’ 2015-04-25 14:00:31 5821 [Note] Shutting down plugin ‘XTRADB_INTERNAL_HASH_TABLES’ 2015-04-25 14:00:31 5821 [Note] Shutting down plugin ‘XTRADB_READ_VIEW’ 2015-04-25 14:00:31 5821 [Note] Shutting down plugin ‘BLACKHOLE’ 2015-04-25 14:00:31 5821 [Note] Shutting down plugin ‘PERFORMANCE_SCHEMA’ 2015-04-25 14:00:31 5821 [Note] Shutting down plugin ‘MEMORY’ 2015-04-25 14:00:31 5821 [Note] Shutting down plugin ‘MRG_MYISAM’ 2015-04-25 14:00:31 5821 [Note] Shutting down plugin ‘CSV’ 2015-04-25 14:00:31 5821 [Note] Shutting down plugin ‘MyISAM’ 2015-04-25 14:00:31 5821 [Note] Shutting down plugin ‘sha256_password’ 2015-04-25 14:00:31 5821 [Note] Shutting down plugin ‘mysql_old_password’ 2015-04-25 14:00:31 5821 [Note] Shutting down plugin ‘mysql_native_password’ 2015-04-25 14:00:31 5821 [Note] Shutting down plugin ‘wsrep’ 2015-04-25 14:00:31 5821 [Note] Shutting down plugin ‘binlog’ 2015-04-25 14:00:31 5821 [Note] /usr/sbin/mysqld: Shutdown complete

So this instance failed due to non initialized InnoDB structures:


2015-04-25 14:00:31 5821 [ERROR] InnoDB: space header page consists of zero bytes in data file ./ibdata1
2015-04-25 14:00:31 5821 [ERROR] InnoDB: Could not open or create the system tablespace. If you tried to add new data files to the system tablespace, and it failed here, you should now edit innodb_data_file_path in my.cnf back to what it was, and remove the new ibdata files InnoDB created in this failed attempt. InnoDB only wrote those files full of zeros, but did not yet use them in any way. But be careful: do not remove old data files which contain your precious data!
2015-04-25 14:00:31 5821 [ERROR] Plugin 'InnoDB' init function returned error.
2015-04-25 14:00:31 5821 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.

what if you use:
[HTML]mysql_install_db[/HTML]
on the joiner node before starting?