Not the answer you need?
Register and ask your own question!

SST XtraDB Backup - Use a different .sst directory for JOINER

HaroldHarold EntrantCurrent User Role Beginner
Hi there !

I'm trying to restore one node of my percona cluster. Usually, everything goes well.
Unfortunatly, I'm running out of space into my datadir : XtraBackup create a ".sst" directory under datadir that fill with backup file

See log output :

WSREP_SST: [INFO] WARNING: Stale temporary SST directory: /datas/mysql//.sst from previous state transfer. Removing
WSREP_SST: [INFO] Proceeding with SST (20170404 10:40:00.090)
WSREP_SST: [INFO] Evaluating socat -u TCP-LISTEN:4444,reuseaddr stdio | xbstream -x; RC=( ${PIPESTATUS[@]} ) (20170404 10:40:00.091)
WSREP_SST: [INFO] Cleaning the existing datadir and innodb-data/log directories (20170404 10:40:00.093)
WSREP_SST: [INFO] Waiting for SST streaming to complete! (20170404 10:40:00.119)
2017-04-04 10:40:00 16965 [Note] WSREP: (485f6f57, 'tcp://0.0.0.0:4567') turning message relay requesting off
2017-04-04 10:54:16 16965 [Note] WSREP: 0.0 (node_cluster_56): State transfer to 1.0 (node_cluster_56) complete.
2017-04-04 10:54:16 16965 [Note] WSREP: Member 0.0 (node_cluster_56) synced with group.
WSREP_SST: [INFO] Compressed qpress files found (20170404 10:54:16.800)
WSREP_SST: [INFO] Decompression with 32 threads (20170404 10:54:16.806)
WSREP_SST: [INFO] Evaluating find /datas/mysql//.sst -type f -name '*.qp' -printf '%p\n%h\n' | xargs -n 2 qpress -T32d (20170404 10:54:16.810)
^M^M ^M^Mqpress: Disk full while writing destination file
xargs: qpress : a terminé son exécution avec le statut 255 ; arrêt abrupt.
WSREP_SST: [ERROR] Cleanup after exit with status:124 (20170404 11:13:27.825)
2017-04-04 11:13:27 16965 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'joiner' --address '10.1.1.8' --datadir '/datas/mysql/' --defaults-file '/etc/mysql/my.cnf' --defaults-group-suffix '' --parent '16965' '' : 124 (Wrong medium type)
2017-04-04 11:13:27 16965 [ERROR] WSREP: Failed to read uuid:seqno from joiner script.
2017-04-04 11:13:27 16965 [ERROR] WSREP: SST script aborted with error 124 (Wrong index given to function)
2017-04-04 11:13:27 16965 [ERROR] WSREP: SST failed: 124 (Wrong medium type)
2017-04-04 11:13:27 16965 [ERROR] Aborting

Unfortunatly (again), you can't not simply change the .sst directoty for the joiner in wsrep_sst_xtrabackup-v2 script.
So I tried to "hack" it to create my own .sst directory who is symbolic link to antoher partition.

rm -rf /datas/mysql/*
rm -rf /datas/mysql/.sst
ln -s /var/tmp/xtrabackup /datas/mysql/.sst
chown -R mysql:mysql /datas/mysql
chown -R mysql:mysql /var/tmp/xtrabackup

It's works until innobackupex state :

WSREP_SST: [INFO] Preparing the backup at /datas/mysql//.sst (20170405 10:52:46.812)
WSREP_SST: [INFO] Evaluating innobackupex --no-version-check --apply-log $rebuildcmd ${DATA} &>${DATA}/innobackup.prepare.log (20170405 10:52:46.816)
WSREP_SST: [ERROR] Cleanup after exit with status:1 (20170405 10:52:46.861)
2017-04-05 10:52:46 30211 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'joiner' --address '192.168.118.8' --datadir '/datas/mysql/' --defaults-file '/etc/mysql/my.cnf' --defaults-group-suffix '' --parent '30211' '' : 1 (Operation
not permitted)
2017-04-05 10:52:46 30211 [ERROR] WSREP: Failed to read uuid:seqno from joiner script.
2017-04-05 10:52:46 30211 [ERROR] WSREP: SST script aborted with error 1 (Operation not permitted)
2017-04-05 10:52:46 30211 [ERROR] WSREP: SST failed: 1 (Operation not permitted)
2017-04-05 10:52:46 30211 [ERROR] Aborting

More details in innobackup.prepare.log :

170405 11:32:21 innobackupex: Starting the apply-log operation

IMPORTANT: Please check that the apply-log run completes successfully.
At the end of a successful apply-log run innobackupex
prints "completed OK!".

innobackupex version 2.3.6 based on MySQL server 5.6.24 Linux (x86_64) (revision id: 7686bfc)
xtrabackup: cd to /datas/mysql//.sst
xtrabackup: This target seems to be not prepared yet.
2017-04-05 11:32:21 7f74d64907e0 InnoDB: Operating system error number 2 in a file operation.
InnoDB: The error means the system cannot find the path specified.
xtrabackup: Warning: cannot open ./xtrabackup_logfile. will try to find.
2017-04-05 11:32:21 7f74d64907e0 InnoDB: Operating system error number 2 in a file operation.
InnoDB: The error means the system cannot find the path specified.
xtrabackup: Fatal error: cannot find ./xtrabackup_logfile.
xtrabackup: Error: xtrabackup_init_temp_log() failed.

Another way is to avoid SST :
https://severalnines.com/blog/how-avoid-sst-when-adding-new-node-galera-cluster

It not seems to complex but in production envireonnement, I'm not confortable with it.

Has anyone faced that issue ? Have other suggestions ?

Regards,

Comments

  • vadimtkvadimtk Contributor Percona Staff Role
    I see you are probably using compression for SST.
    Can you disable compression and see if it works for you?
  • HaroldHarold Entrant Current User Role Beginner
    Hi vadimtk,

    You are right, I'm using compression for SST. Thank for your suggestion, I gonna try it.

    As far I have a space issue, I have also tried to set my datadir in another partition with higher space. That action bring another issue :
    2017-04-11 10:59:19 9082 [Warning] WSREP: Gap in state sequence. Need state transfer.
    2017-04-11 10:59:19 9082 [Note] WSREP: Running: 'wsrep_sst_xtrabackup-v2 --role 'joiner' --address '192.168.118.8' --datadir '/datas/mysql/' --defaults-file '/etc/mysql/my.cnf' --defaults-group-suffix '' --parent '9082' '' '
    WSREP_SST: [INFO] Streaming with xbstream (20170411 10:59:19.756)
    WSREP_SST: [INFO] Using socat as streamer (20170411 10:59:19.760)
    WSREP_SST: [INFO] Evaluating timeout -s9 100 socat -u TCP-LISTEN:4444,reuseaddr stdio | xbstream -x; RC=( ${PIPESTATUS[@]} ) (20170411 10:59:19.820)
    2017-04-11 10:59:19 9082 [Note] WSREP: Prepared SST request: xtrabackup-v2|192.168.118.8:4444/xtrabackup_sst//1
    2017-04-11 10:59:19 9082 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
    2017-04-11 10:59:19 9082 [Note] WSREP: REPL Protocols: 7 (3, 2)
    2017-04-11 10:59:19 9082 [Note] WSREP: Assign initial position for certification: 3332615043, protocol version: 3
    2017-04-11 10:59:19 9082 [Note] WSREP: Service thread queue flushed.
    2017-04-11 10:59:19 9082 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (ca9ef4d7-1407-11e5-9d79-a6e0476b8926): 1 (Operation not permitted)
    at galera/src/replicator_str.cpp:prepare_for_IST():535. IST will be unavailable.
    2017-04-11 10:59:19 9082 [Note] WSREP: Member 1.0 (node_cluster_56) requested state transfer from '*any*'. Selected 0.0 (node_cluster_56)(SYNCED) as donor.
    2017-04-11 10:59:19 9082 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 3332615054)
    2017-04-11 10:59:19 9082 [Note] WSREP: Requesting state transfer: success, donor: 0
    2017-04-11 10:59:19 9082 [Note] WSREP: GCache history reset: old(00000000-0000-0000-0000-000000000000:0) -> new(ca9ef4d7-1407-11e5-9d79-a6e0476b8926:3332615043)
    2017-04-11 10:59:20 9082 [Warning] WSREP: 0.0 (node_cluster_56): State transfer to 1.0 (node_cluster_56) failed: -1 (Operation not permitted)
    2017-04-11 10:59:20 9082 [ERROR] WSREP: gcs/src/gcs_group.cpp:gcs_group_handle_join_msg():765: Will never receive state. Need to abort.

    2017-04-11 10:59:20 9082 [Note] WSREP: gcomm: terminating thread
    2017-04-11 10:59:20 9082 [Note] WSREP: gcomm: joining thread
    2017-04-11 10:59:20 9082 [Note] WSREP: gcomm: closing backend
    2017-04-11 10:59:20 9082 [Note] WSREP: view(view_id(NON_PRIM,11238dec,716) memb {
    258aefce,0
    } joined {
    } left {
    } partitioned {
    11238dec,0
    7d14391a,0
    ad7f0066,0
    c716ce0b,0
    })
    2017-04-11 10:59:20 9082 [Note] WSREP: view((empty))
    2017-04-11 10:59:20 9082 [Note] WSREP: gcomm: closed
    2017-04-11 10:59:20 9082 [Note] WSREP: /usr/sbin/mysqld: Terminated.
    170411 10:59:20 mysqld_safe mysqld from pid file /datas/mysql/p2-mysql08.pid ended

    Maybe an error from DONOR side ?
  • przemekprzemek Percona Support Engineer Percona Staff Role
    Hi,

    At a stage when a donor was already selected, yes, it is important to check also it's corresponding error log entries. If produced, check also the innobackup.backup.log on donor.
  • HaroldHarold Entrant Current User Role Beginner
    Hi przemek,

    I can see the same error on the DONOR (the cluster master is always selected as :( ) . Unfortunaltly, there are no more details

    2017-04-11 10:11:22 27232 [Note] WSREP: Member 1.0 (node_cluster_56) requested state transfer from '*any*'. Selected 0.0 (node_cluster_56)(SYNCED) as donor.
    2017-04-11 10:11:22 27232 [Note] WSREP: Shifting SYNCED -> DONOR/DESYNCED (TO: 3332581902)
    2017-04-11 10:11:22 27232 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
    2017-04-11 10:11:22 27232 [Note] WSREP: Running: 'wsrep_sst_xtrabackup-v2 --role 'donor' --address '192.168.118.8:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/datas/mysql/' --defaults-file '/etc/mysql/my.cnf' --defaults-group-suffix '' '' --gtid 'ca9ef4d7-1407-11e5-9d79-a6e0476b8926:3332581902''
    2017-04-11 10:11:22 27232 [Note] WSREP: sst_donor_thread signaled with 0
    2017-04-11 10:11:23 27232 [ERROR] WSREP: Failed to read from: wsrep_sst_xtrabackup-v2 --role 'donor' --address '192.168.118.8:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/datas/mysql/' --defaults-file '/etc/mysql/my.cnf' --defaults-group-suffix ''
    '' --gtid 'ca9ef4d7-1407-11e5-9d79-a6e0476b8926:3332581902'

    2017-04-11 10:11:23 27232 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'donor' --address '192.168.118.8:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/datas/mysql/' --defaults-file '/etc/mysql/my.cnf' --defaults-group-suffix '' '' --gtid 'ca9ef4d7-1407-11e5-9d79-a6e0476b8926:3332581902': 1 (Operation not permitted)
    2017-04-11 10:11:23 27232 [ERROR] WSREP: Command did not run: wsrep_sst_xtrabackup-v2 --role 'donor' --address '192.168.118.8:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/datas/mysql/' --defaults-file '/etc/mysql/my.cnf' --defaults-group-suffix ''
    '' --gtid 'ca9ef4d7-1407-11e5-9d79-a6e0476b8926:3332581902'
    2017-04-11 10:11:23 27232 [Warning] WSREP: 0.0 (node_cluster_56): State transfer to 1.0 (node_cluster_56) failed: -1 (Operation not permitted)
    2017-04-11 10:11:23 27232 [Note] WSREP: Shifting DONOR/DESYNCED -> JOINED (TO: 3332581905)
    2017-04-11 10:11:23 27232 [Note] WSREP: Member 0.0 (node_cluster_56) synced with group.

    I am currently re-installing Percona XtraBD for the damaged node and I will do a uncompress SST.

    The file innobackup.backup.log has been overwritten
  • przemekprzemek Percona Support Engineer Percona Staff Role
    Hi,

    This looks like SELInux or Apparmor issue to me. The log says the sst script wasn't even executed on the donor. Please check the corresponding security logs.
  • HaroldHarold Entrant Current User Role Beginner
    Hi przemek,

    I'm under CentOS 6 and SELInux is disable.
    I have finally re-installed PXC on this node and I have succed to joined a QA clucter with it.
    We gonna re-try with PROD env now.
  • HaroldHarold Entrant Current User Role Beginner
    Hi,

    With PXC 5.6.35-20-3, you can define now the tmpdir for the backup durring the SST

    https://jira.percona.com/browse/PXC-782

    https://www.percona.com/blog/2017/04/19/percona-xtradb-cluster-5-6-35-26-20-3-now-available/
  • HaroldHarold Entrant Current User Role Beginner
    Hi,

    Fianlly, We have purged a huge database and we retrived space. We succed to restore our lost node.
    Thank to the community for his help

    Best regards,

    Harold
Sign In or Register to comment.

MySQL, InnoDB, MariaDB and MongoDB are trademarks of their respective owners.
Copyright ©2005 - 2020 Percona LLC. All rights reserved.