Not the answer you need?
Register and ask your own question!

Cannot join second node

SouthernBelleSouthernBelle ContributorInactive User Role Beginner
I have been trying to set up a new XtraDB Cluster like several times before, following this HOWTO: http://www.percona.com/doc/percona-x...ots_howto.html.

After setting up the first node with an appropriate user for sst, I've now been trying to start the second node for a day. On the donor side, mysql.log and innobackup.backup.log look perfectly fine, but the joiner mysql.log tells me that xtrabackup_galera_info wasn't created, and wsrep_sst_xtrabackup terminates with code 32 (broken pipe). I have searched all archives I could get hold of, but could not find anything appropriate. Most people are pointing to error messages in the innobackup.backup.log, but this really seems to be clean in my case. Does anybody have any idea? I would be really extremely grateful for any hint.

Comments

  • SouthernBelleSouthernBelle Contributor Inactive User Role Beginner
    I really can't convince innobackupex to create galera_info. The call is

    Where should I expect galera_info? In /tmp maybe or /var/lib/mysql? Nothing... :(

    [DRUPAL-LIVE] [email protected] sudo -u mysql innobackupex --defaults-file=/etc/my.cnf --user=sstuser --password=UmRgRe77sjOmshuXm8odtt --socket=/var/lib/mysql/mysql.sock --galera-info --stream=tar /tmp >/tmp/innobackupex.out

    InnoDB Backup Utility v1.5.1-xtrabackup; Copyright 2003, 2009 Innobase Oy
    and Percona LLC and/or its affiliates 2009-2013. All Rights Reserved.

    This software is published under
    the GNU GENERAL PUBLIC LICENSE Version 2, June 1991.

    Get the latest version of Percona XtraBackup, documentation, and help resources:
    http://www.percona.com/xb/p

    141210 14:29:35 innobackupex: Connecting to MySQL server with DSN 'dbi:mysql:;mysql_read_default_file=/etc/my.cnf;mysql_read_default_group=xtrabackup;mysql_socket=/var/lib/mysql/mysql.sock' as 'sstuser' (using password: YES).
    141210 14:29:35 innobackupex: Connected to MySQL server
    141210 14:29:35 innobackupex: Executing a version check against the server...
    141210 14:29:35 innobackupex: Done.
    141210 14:29:35 innobackupex: Starting the backup operation

    IMPORTANT: Please check that the backup run completes successfully.
    At the end of a successful backup run innobackupex
    prints "completed OK!".

    innobackupex: Using server version 5.6.21-70.1-56-log

    innobackupex: Created backup directory /tmp

    141210 14:29:35 innobackupex: Starting ibbackup with command: xtrabackup --defaults-file="/etc/my.cnf" --defaults-group="mysqld" --backup --suspend-at-end --target-dir=/tmp --tmpdir=/tmp --extra-lsndir='/tmp' --stream=tar
    innobackupex: Waiting for ibbackup (pid=15401) to suspend
    innobackupex: Suspend file '/tmp/xtrabackup_suspended_2'

    xtrabackup version 2.2.6 based on MySQL server 5.6.21 Linux (x86_64) (revision id: )
    xtrabackup: uses posix_fadvise().
    xtrabackup: cd to /var/lib/mysql
    xtrabackup: open files limit requested 0, set to 1024000
    xtrabackup: using the following InnoDB configuration:
    xtrabackup: innodb_data_home_dir = ./
    xtrabackup: innodb_data_file_path = ibdata1:10M:autoextend
    xtrabackup: innodb_log_group_home_dir = ./
    xtrabackup: innodb_log_files_in_group = 3
    xtrabackup: innodb_log_file_size = 268435456
    >> log scanned up to (1626850)
    xtrabackup: Generating a list of tablespaces
    [01] Streaming ./ibdata1
    [01] ...done
    [01] Streaming ./mysql/slave_worker_info.ibd
    [01] ...done
    [01] Streaming ./mysql/innodb_table_stats.ibd
    [01] ...done
    [01] Streaming ./mysql/innodb_index_stats.ibd
    [01] ...done
    [01] Streaming ./mysql/slave_relay_log_info.ibd
    [01] ...done
    [01] Streaming ./mysql/slave_master_info.ibd
    [01] ...done
    >> log scanned up to (1626850)
    xtrabackup: Creating suspend file '/tmp/xtrabackup_suspended_2' with pid '15401'

    141210 14:29:37 innobackupex: Continuing after ibbackup has suspended
    141210 14:29:37 innobackupex: Executing LOCK TABLES FOR BACKUP...
    141210 14:29:37 innobackupex: Backup tables lock acquired

    141210 14:29:37 innobackupex: Starting to backup non-InnoDB tables and files
    innobackupex: in subdirectories of '/var/lib/mysql/'
    innobackupex: Backing up files '/var/lib/mysql//performance_schema/*.{frm,isl,MYD,MYI,MAD,MAI,MRG,TRG,TRN,ARM,ARZ,CSM,CSV,opt,par}' (53 files)
    >> log scanned up to (1626850)
    innobackupex: Backing up files '/var/lib/mysql//mysql/*.{frm,isl,MYD,MYI,MAD,MAI,MRG,TRG,TRN,ARM,ARZ,CSM,CSV,opt,par}' (74 files)
    141210 14:29:38 innobackupex: Finished backing up non-InnoDB tables and files

    141210 14:29:38 innobackupex: Executing LOCK BINLOG FOR BACKUP...
    141210 14:29:38 innobackupex: Executing FLUSH ENGINE LOGS...
    141210 14:29:38 innobackupex: Waiting for log copying to finish

    xtrabackup: The latest check point (for incremental): '1626850'
    xtrabackup: Stopping log copying thread.
    .>> log scanned up to (1626850)

    xtrabackup: Creating suspend file '/tmp/xtrabackup_log_copied' with pid '15401'
    xtrabackup: Transaction log of lsn (1626850) to (1626850) was copied.
    141210 14:29:39 innobackupex: Executing UNLOCK BINLOG
    141210 14:29:39 innobackupex: Executing UNLOCK TABLES
    141210 14:29:39 innobackupex: All tables unlocked

    innobackupex: Backup created in directory '/tmp'
    innobackupex: MySQL binlog position: filename 'mysql-bin.000006', position 875, GTID of the last change ''
    141210 14:29:39 innobackupex: Connection to database server closed
    innobackupex: You must use -i (--ignore-zeros) option for extraction of the tar stream.
    141210 14:29:39 innobackupex: completed OK!
  • SouthernBelleSouthernBelle Contributor Inactive User Role Beginner
    On the joiner, the following files are created:

    -rw-rw---- 1 mysql mysql 23 Dec 10 16:53 xtrabackup_binlog_info
    -rw-rw---- 1 root root 89 Dec 10 16:53 xtrabackup_checkpoints
    -rw-rw---- 1 mysql mysql 675 Dec 10 16:53 xtrabackup_info
    -rw-rw---- 1 root root 2.5K Dec 10 16:53 xtrabackup_logfile

    In my working installations of XtraDB Cluster, there's also xtrabackup_galera_info, the one that's apparently not being created here.

    Strangely, some of the files belong to root, others to the mysql user.
  • SouthernBelleSouthernBelle Contributor Inactive User Role Beginner
    I reinstalled PXC from scratch, removed the old datadirs and used wsrep_sst_method=xtrabackup-v2 this time. I followed http://www.percona.com/doc/percona-xtradb-cluster/5.6/howtos/ubuntu_howto.html step by step.

    The joiner does not come up, and the log is full with messages like

    ^G/usr/sbin/mysqld: Can't find file: './mysql/plugin.frm' (errno: 13 - Permission denied)
    2014-12-11 02:10:41 29779 [ERROR] Can't open the mysql.plugin table. Please run mysql_upgrade to create it.
    2014-12-11 02:10:41 29779 [Note] InnoDB: Using atomics to ref count buffer pool pages
    2014-12-11 02:10:41 29779 [Note] InnoDB: The InnoDB memory heap is disabled
    2014-12-11 02:10:41 29779 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
    2014-12-11 02:10:41 29779 [Note] InnoDB: Memory barrier is not used
    2014-12-11 02:10:41 29779 [Note] InnoDB: Compressed tables use zlib 1.2.3
    2014-12-11 02:10:41 29779 [Note] InnoDB: Using Linux native AIO
    2014-12-11 02:10:41 29779 [Note] InnoDB: Using CPU crc32 instructions
    2014-12-11 02:10:41 29779 [Note] InnoDB: Initializing buffer pool, size = 4.0G
    2014-12-11 02:10:41 29779 [Note] InnoDB: Completed initialization of buffer pool
    2014-12-11 02:10:41 29779 [Note] InnoDB: Highest supported file format is Barracuda.
    2014-12-11 02:10:41 7f6a5f35f800 InnoDB: Operating system error number 13 in a file operation.
    InnoDB: The error means mysqld does not have the access rights to
    InnoDB: the directory.
    2014-12-11 02:10:41 29779 [ERROR] InnoDB: Could not find a valid tablespace file for 'mysql/innodb_index_stats'. See http://dev.mysql.com/doc/refman/5.6/...-datadict.html for how to resolve the issue.
    2014-12-11 02:10:41 29779 [ERROR] InnoDB: Tablespace open failed for '"mysql"."innodb_index_stats"', ignored.
    2014-12-11 02:10:41 7f6a5f35f800 InnoDB: Operating system error number 13 in a file operation.
    InnoDB: The error means mysqld does not have the access rights to
    InnoDB: the directory.
    2014-12-11 02:10:41 29779 [ERROR] InnoDB: Could not find a valid tablespace file for 'mysql/innodb_table_stats'. See http://dev.mysql.com/doc/refman/5.6/...-datadict.html for how to resolve the issue.


    Some of the content in the datadir on the joiner does in fact belong to root:
    [LIVE] [email protected] ls -l /var/lib/mysql/
    total 973M
    -rw-rw---- 1 mysql mysql 56 Dec 11 02:10 auto.cnf
    -rw-rw---- 1 root root 359 Dec 11 02:10 backup-my.cnf
    -rw
    1 mysql mysql 129M Dec 11 02:10 galera.cache
    -rw-rw---- 1 mysql mysql 104 Dec 11 02:10 grastate.dat
    -rw-rw---- 1 mysql mysql 217 Dec 11 02:10 gvwstate.dat
    -rw-rw---- 1 root root 962 Dec 11 02:10 ib_buffer_pool
    -rw-rw---- 1 root root 74M Dec 11 02:10 ibdata1
    -rw-rw---- 1 root root 256M Dec 11 02:10 ib_logfile0
    -rw-rw---- 1 root root 256M Dec 11 02:10 ib_logfile1
    -rw-rw---- 1 root root 256M Dec 11 02:10 ib_logfile2
    -rw-rw---- 1 root root 5.5K Dec 11 02:10 innobackup.prepare.log
    drwx
    2 root root 4.0K Dec 11 02:10 mysql
    -rw-rw---- 1 mysql mysql 120 Dec 11 02:10 mysql-bin.000001
    -rw-rw---- 1 mysql mysql 19 Dec 11 02:10 mysql-bin.index
    drwx
    2 root root 4.0K Dec 11 02:10 performance_schema
    -rw-r--r-- 1 root root 118 Dec 11 02:07 RPM_UPGRADE_HISTORY
    drwx
    2 root root 4.0K Dec 11 02:10 test
    -rw-rw---- 1 root root 21 Dec 11 02:10 xtrabackup_binlog_info
    -rw-rw---- 1 root root 89 Dec 11 02:10 xtrabackup_checkpoints
    -rw-rw---- 1 root root 38 Dec 11 02:10 xtrabackup_galera_info
    -rw-rw---- 1 root root 720 Dec 11 02:10 xtrabackup_info
    -rw-rw---- 1 root root 2.0M Dec 11 02:10 xtrabackup_logfile

    This is a new installation, and the user in my.cnf is mysql, not root. I do start the init script as root, but I think this is normal.
  • przemekprzemek Percona Support Engineer Percona Staff Role
    How did you prepare the joiner before starting with wsrep-enabled my.cnf? Is the /var/lib/mysql/ owned by mysql user and group?
    Also if you do restore backup manually from donor, remember to chown all files to mysql user/group too.
    Another thing is when you initialize datadir with mysql_install_db, use the "--user=mysql" option.
  • SouthernBelleSouthernBelle Contributor Inactive User Role Beginner
    Thank you for your reply. I got the cluster to work with an older version. Now it also works again with the latest version available on the yum repo. Apparently it was an issue with a build that was out for a couple of days / weeks.
Sign In or Register to comment.

MySQL, InnoDB, MariaDB and MongoDB are trademarks of their respective owners.
Copyright ©2005 - 2020 Percona LLC. All rights reserved.