Not the answer you need?
Register and ask your own question!

Nodes terminated when addeted new

cyber_neocyber_neo EntrantInactive User Role Beginner
Hello!

I have cluster of 2 nodes (master-master). One of them master for slave.
When i tried add new master node in cluster Donor node and acceptor fall down. On donor node in innobackup.backup.log i see next:
150121 14:49:11 innobackupex: Finished backing up non-InnoDB tables and files

150121 14:49:11 innobackupex: Executing LOCK BINLOG FOR BACKUP...
DBD::mysql::db do failed: Deadlock found when trying to get lock; try restarting transaction at /usr//bin/innobackupex line 3036.
innobackupex: got a fatal error with the following stacktrace: at /usr//bin/innobackupex line 3039
main::mysql_query('HASH(0x10a9720)', 'LOCK BINLOG FOR BACKUP') called at /usr//bin/innobackupex line 3501
main::mysql_lock_binlog('HASH(0x10a9720)') called at /usr//bin/innobackupex line 2000
main::backup() called at /usr//bin/innobackupex line 1592
innobackupex: Error:
Error executing 'LOCK BINLOG FOR BACKUP': DBD::mysql::db do failed: Deadlock found when trying to get lock; try restarting transaction at /usr//bin/innobackupex line 3036.
150121 14:49:11 innobackupex: Waiting for ibbackup (pid=44712) to finish


Versions:
mysqld Ver 5.6.21-70.1-56 for Linux on x86_64 (Percona XtraDB Cluster (GPL), Release rel70.1, Revision 938, WSREP version 25.8, wsrep_25.8.r4150)

xtrabackup version 2.2.8 based on MySQL server 5.6.22 Linux (x86_64) (revision id: )



I tried to setup new node and got same error.

Comments

  • przemekprzemek Percona Support Engineer Percona Staff Role
    You may be hitting this bug: https://bugs.launchpad.net/percona-xtradb-cluster/+bug/1401133
    Did mysql also crash like mentioned in the bug report?
  • XibuXibu Contributor Inactive User Role Beginner
    I got the same issue since March 11, when I installed the new Percona version
    Mar 11 10:51:32 Updated: Percona-XtraDB-Cluster-galera-3-3.9-1.3494.rhel6.x86_64
    Mar 11 10:51:33 Updated: 1:Percona-XtraDB-Cluster-client-56-5.6.22-25.8.978.el6.x86_64
    Mar 11 10:51:35 Updated: percona-xtrabackup-2.2.9-5067.el6.x86_64
    Mar 11 10:51:35 Updated: 1:Percona-XtraDB-Cluster-shared-56-5.6.22-25.8.978.el6.x86_64
    Mar 11 10:52:14 Updated: 1:Percona-XtraDB-Cluster-server-56-5.6.22-25.8.978.el6.x86_64
    Mar 11 10:52:14 Updated: 1:Percona-XtraDB-Cluster-56-5.6.22-25.8.978.el6.x86_64

    my backup crash every night over the same point

    mysql error log
    ==================

    2015-03-23 16:06:47 3443 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 122394644)
    2015-03-23 16:06:47 3443 [Note] WSREP: Synchronized with group, ready for connections
    2015-03-23 16:06:50 3443 [Note] WSREP: (045575ff, 'tcp://0.0.0.0:4567') turning message relay requesting off
    03:50:04 UTC - mysqld got signal 11 ;
    This could be because you hit a bug. It is also possible that this binary
    or one of the libraries it was linked against is corrupt, improperly built,
    or misconfigured. This error can also be caused by malfunctioning hardware.
    We will try our best to scrape up some info that will hopefully help
    diagnose the problem, but since we have already crashed,
    something is definitely wrong and this may fail.
    Please help us make Percona XtraDB Cluster better by reporting any
    bugs at https://bugs.launchpad.net/percona-xtradb-cluster

    key_buffer_size=8388608
    read_buffer_size=131072
    max_used_connections=634
    max_threads=2050
    thread_count=18
    connection_count=1
    It is possible that mysqld could use up to
    key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 826591 K bytes of memory
    Hope that's ok; if not, decrease some variables in the equation.

    Thread pointer: 0x65f8e9f0
    Attempting backtrace. You can use the following information to find out
    where mysqld died. If you see no messages after this, something went
    terribly wrong...
    stack_bottom = 7f00a16b8d38 thread_stack 0x40000
    /usr/sbin/mysqld(my_print_stacktrace+0x35)[0x8fa965]
    /usr/sbin/mysqld(handle_fatal_signal+0x4b4)[0x665644]
    /lib64/libpthread.so.0(+0xf710)[0x7f1812ff0710]
    [0x7eff580000c8]

    Trying to get some variables.
    Some pointers may be invalid and cause the dump to abort.
    Query (0): is an invalid pointer
    Connection ID (thread ID): 240147
    Status: KILL_CONNECTION

    You may download the Percona XtraDB Cluster operations manual by visiting
    http://www.percona.com/software/percona-xtradb-cluster/. You may find information
    in the manual which will help you identify the cause of the crash.
    150326 04:50:06 mysqld_safe Number of processes running now: 0
    150326 04:50:06 mysqld_safe WSREP: not restarting wsrep node automatically
    150326 04:50:06 mysqld_safe mysqld from pid file /var/lib/mysql/XXXXXXXXXXXXXXX.pid ended

    mysql backup log
    ==============

    >> log scanned up to (419166230342)
    >> log scanned up to (419166242232)
    [04] ...done
    xtrabackup: Creating suspend file 'XXXXXXXXXXXXXX/xtrabackup/tempbackup/xtrabackup_suspended_2' with pid '834130'
    >> log scanned up to (419166256114)

    150326 04:49:56 innobackupex: Continuing after ibbackup has suspended
    150326 04:49:56 innobackupex: Executing LOCK TABLES FOR BACKUP...
    150326 04:49:56 innobackupex: Backup tables lock acquired

    150326 04:49:56 innobackupex: Starting to backup non-InnoDB tables and files
    innobackupex: in subdirectories of '/var/lib/mysql/'
    innobackupex: Backing up files '/var/lib/mysql//mysql/*.{frm,isl,MYD,MYI,MAD,MAI,MRG,TRG,TRN,ARM,ARZ,CSM,CSV,opt,par}' (74 files)
    >> log scanned up to (419166337626)
    >> log scanned up to (419166341849)
    innobackupex: Backing up files '/var/lib/mysql//XXXXXXX/*.{frm,isl,MYD,MYI,MAD,MAI,MRG,TRG,TRN,ARM,ARZ,CSM,CSV,opt,par}' (69 files)
    >> log scanned up to (419166341849)
    >> log scanned up to (419166341849)
    >> log scanned up to (419166341849)
    >> log scanned up to (419166341849)
    innobackupex: Backing up file '/var/lib/mysql//wsrep/db.opt'
    innobackupex: Backing up file '/var/lib/mysql//wsrep/membership.frm'
    innobackupex: Backing up file '/var/lib/mysql//wsrep/status.frm'
    innobackupex: Backing up files '/var/lib/mysql//XXXXXXX/*.{frm,isl,MYD,MYI,MAD,MAI,MRG,TRG,TRN,ARM,ARZ,CSM,CSV,opt,par}' (119 files)
    >> log scanned up to (419166341849)
    >> log scanned up to (419166341849)
    innobackupex: Backing up files '/var/lib/mysql//performance_schema/*.{frm,isl,MYD,MYI,MAD,MAI,MRG,TRG,TRN,ARM,ARZ,CSM,CSV,opt,par}' (53 files)
    150326 04:50:04 innobackupex: Finished backing up non-InnoDB tables and files

    150326 04:50:04 innobackupex: Executing LOCK BINLOG FOR BACKUP...
    DBD::mysql::db do failed: Deadlock found when trying to get lock; try restarting transaction at /usr/bin/innobackupex line 3045.
    innobackupex: got a fatal error with the following stacktrace: at /usr/bin/innobackupex line 3048
    main::mysql_query('HASH(0x1843178)', 'LOCK BINLOG FOR BACKUP') called at /usr/bin/innobackupex line 3517
    main::mysql_lock_binlog('HASH(0x1843178)') called at /usr/bin/innobackupex line 2009
    main::backup() called at /usr/bin/innobackupex line 1601
    innobackupex: Error:
    Error executing 'LOCK BINLOG FOR BACKUP': DBD::mysql::db do failed: Deadlock found when trying to get lock; try restarting transaction at /usr/bin/innobackupex line 3045.
    150326 04:50:04 innobackupex: Waiting for ibbackup (pid=834130) to finish
    03/26/2015 04:51:50

    The only way I found is to stop the second mysql node of the cluster during the backup

    Hope you could provide us with a solution quickly.
  • przemekprzemek Percona Support Engineer Percona Staff Role
    Xibu,

    Yes, there is a workaround where you may disable the LOCK BINLOG FOR BACKUP feature, so old way FTWRL will be used instead.
    You have to upgrade Percona Xtrabackup to at least 2.2.9 and add this to the my.cnf:
    [sst]
    inno-backup-opts='--no-backup-locks'

    This was mentioned in comment #7 in https://bugs.launchpad.net/percona-xtradb-cluster/+bug/1401133
Sign In or Register to comment.

MySQL, InnoDB, MariaDB and MongoDB are trademarks of their respective owners.
Copyright ©2005 - 2020 Percona LLC. All rights reserved.