# innobackupex does not seem to read datadir correctly **URL:** \ **Category:** Percona XtraBackup\ **Created:** [January 4, 2016, 1:51pm UTC](https://forums.percona.com/t/innobackupex-does-not-seem-to-read-datadir-correctly/4643 "2016-01-04T13:51:50Z")\ **Posts on this page:** 2\ **Page:** 1
**Author:** ![hvo408](https://avatars.discourse-cdn.com/v4/letter/h/ecd19e/32.png) [@hvo408](https://forums.percona.com/u/hvo408)\ **Post date:** [January 4, 2016, 1:51pm UTC](https://forums.percona.com/t/innobackupex-does-not-seem-to-read-datadir-correctly/4643/1 "2016-01-04T13:51:50Z")
Hi, I have a cluster setup with just 2 nodes, and I am using xtrabackup-v2 as the wsrep\_sst\_method. Version info is: - mysql Ver 15.1 Distrib 10.1.10-MariaDB, for Linux (x86\_64) using readline 5.1 - xtrabackup version 2.3.3 based on MySQL server 5.6.24 Linux (x86\_64) (revision id: 525ca7d) On node 1, I can bootstrap mysql successfully. On node 2, I am running into problem with the move-back step. From the mariadb.log, I can see that the innobackupex --move-back step is failing. Under the innobackup.move.log, I am seeing a strange error: - innobackupex version 2.3.3 based on MySQL server 5.6.24 Linux (x86\_64) (revision id: 525ca7d) - Error: Move file ib\_logfile0 to ./ib\_logfile0 failed: Destination file exists I looked into the source code of xtrabackup, and it looks to me like the script didn’t read the datadir conf var correctly. It’s supposed to move the ib\_logfile0 to /var/lib/mysql/ib\_logfile0 but instead it seem to read datadir as “./”. The server.conf is located under my /etc/my.cnf.d directory. Here is the output of mysqld --print-defaults mysqld would have been started with the following arguments: –log-error=/var/log/mariadb.log --socket=/var/lib/mysql/mysql.sock --datadir=/var/lib/mysql --binlog\_format=ROW --default\_storage\_engine=innodb --innodb\_autoinc\_lock\_mode=2 --wsrep\_on=ON --innodb\_doublewrite=1 --wsrep\_provider=/usr/lib64/galera/libgalera\_smm.so --wsrep\_provider\_options=gmcast.listen\_addr=tcp://0.0.0.0:4567; ist.recv\_addr=172.20.222.62:4568 --wsrep\_cluster\_address=gcomm://172.20.222.61,172.20.222.62 --wsrep\_sst\_receive\_address=172.20.222.62:4444 --wsrep\_node\_address=172.20.222.62:4567 --wsrep\_cluster\_name=satelliteDatabaseCluster --wsrep\_node\_name=dlo-lab62 --wsrep\_sst\_method=xtrabackup-v2 --wsrep\_sst\_auth=root:Test123 --wsrep\_debug=ON --bind-address=0.0.0.0 --query\_cache\_size=0 =========== mariadb.log WSREP\_SST: [INFO] Streaming with xbstream (20160104 10:57:13.854) WSREP\_SST: [INFO] Using socat as streamer (20160104 10:57:13.855) WSREP\_SST: [INFO] Evaluating timeout -s9 100 socat -u TCP-LISTEN:4444,reuseaddr stdio | xbstream -x; RC=( ${PIPESTATUS[@]} ) (20160104 10:57:13.879) 2016-01-04 10:57:14 140293202687744 [Note] WSREP: Prepared SST request: xtrabackup-v2|172.20.222.62:4444/xtrabackup\_sst//1 2016-01-04 10:57:14 140293202687744 [Note] WSREP: wsrep\_notify\_cmd is not defined, skipping notification. 2016-01-04 10:57:14 140293202687744 [Note] WSREP: REPL Protocols: 7 (3, 2) 2016-01-04 10:57:14 140292979345152 [Note] WSREP: Service thread queue flushed. 2016-01-04 10:57:14 140293202687744 [Note] WSREP: Assign initial position for certification: 352, protocol version: 3 2016-01-04 10:57:14 140292979345152 [Note] WSREP: Service thread queue flushed. 2016-01-04 10:57:14 140293202687744 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (805ae278-b2ac-11e5-8a4f-8b035dfc5817): 1 (Operation not permitted) at galera/src/replicator\_str.cpp:prepare\_for\_IST():456. IST will be unavailable. 2016-01-04 10:57:14 140292910999296 [Note] WSREP: Member 1.0 (dlo-lab62) requested state transfer from ‘_any_’. Selected 0.0 (dlo-lab61)(SYNCED) as donor. 2016-01-04 10:57:14 140292910999296 [Note] WSREP: Shifting PRIMARY → JOINER (TO: 352) 2016-01-04 10:57:14 140293202687744 [Note] WSREP: Requesting state transfer: success, donor: 0 WSREP\_SST: [INFO] Proceeding with SST (20160104 10:57:14.263) WSREP\_SST: [INFO] Evaluating socat -u TCP-LISTEN:4444,reuseaddr stdio | xbstream -x; RC=( ${PIPESTATUS[@]} ) (20160104 10:57:14.264) WSREP\_SST: [INFO] Cleaning the existing datadir and innodb-data/log directories (20160104 10:57:14.266) removed `/var/lib/mysql/mysql.sock' removed `/var/lib/mysql/ib\_logfile1’ removed `/var/lib/mysql/aria_log.00000001' removed `/var/lib/mysql/aria\_log\_control’ removed `/var/lib/mysql/ibdata1' removed `/var/lib/mysql/ib\_logfile0’ WSREP\_SST: [INFO] Waiting for SST streaming to complete! (20160104 10:57:14.298) 2016-01-04 10:57:16 140292921489152 [Note] WSREP: (f6dbe707, ‘tcp://0.0.0.0:4567’) turning message relay requesting off 2016-01-04 10:57:28 140292910999296 [Note] WSREP: 0.0 (dlo-lab61): State transfer to 1.0 (dlo-lab62) complete. 2016-01-04 10:57:28 140292910999296 [Note] WSREP: Member 0.0 (dlo-lab61) synced with group. WSREP\_SST: [INFO] Preparing the backup at /var/lib/mysql//.sst (20160104 10:57:30.218) WSREP\_SST: [INFO] Evaluating innobackupex --no-version-check --apply-log $rebuildcmd ${DATA} &\>${DATA}/innobackup.prepare.log (20160104 10:57:30.224) rm: cannot remove `/var/lib/mysql//innobackup.prepare.log': No such file or directory rm: cannot remove `/var/lib/mysql//innobackup.move.log’: No such file or directory WSREP\_SST: [INFO] Moving the backup to /var/lib/mysql/ (20160104 10:57:33.948) WSREP\_SST: [INFO] DATA is /var/lib/mysql//.sst (20160104 10:57:33.950) WSREP\_SST: [INFO] Evaluating innobackupex --defaults-file=/etc/my.cnf --no-version-check --move-back --force-non-empty-directories ${DATA} &\> ${DATA}/innobackup.move.log (20160104 10:57:33.951) WSREP\_SST: [ERROR] Cleanup after exit with status:1 (20160104 10:57:33.958) 2016-01-04 10:57:33 140292879546112 [ERROR] WSREP: Process completed with error: wsrep\_sst\_xtrabackup-v2 --role ‘joiner’ --address ‘172.20.222.62:4444’ --datadir ‘/var/lib/mysql/’ --defaults-file ‘/etc/my.cnf’ --parent ‘3840’ ‘’ : 1 (Operation not permitted) 2016-01-04 10:57:33 140292879546112 [ERROR] WSREP: Failed to read uuid:seqno and wsrep\_gtid\_domain\_id from joiner script. 2016-01-04 10:57:33 140293202995232 [ERROR] WSREP: SST failed: 1 (Operation not permitted) 2016-01-04 10:57:33 140293202995232 [ERROR] Aborting Error in my\_thread\_global\_end(): 1 threads didn’t exit =========== innobackup.move.log 160104 10:57:33 innobackupex: Starting the move-back operation IMPORTANT: Please check that the move-back run completes successfully. At the end of a successful move-back run innobackupex prints “completed OK!”. innobackupex version 2.3.3 based on MySQL server 5.6.24 Linux (x86\_64) (revision id: 525ca7d) Error: Move file ib\_logfile0 to ./ib\_logfile0 failed: Destination file exists 18:57:33 UTC - xtrabackup got signal 11 ; This could be because you hit a bug or data is corrupted. This error can also be caused by malfunctioning hardware. We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Thread pointer: 0x0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong… stack\_bottom = 0 thread\_stack 0x10000 innobackupex(my\_print\_stacktrace+0x2e) [0x93827e] innobackupex(handle\_fatal\_signal+0x273) [0x81b483] /lib64/libpthread.so.0(+0xf790) [0x7fded4a86790] /lib64/libc.so.6(cfree+0x1c) [0x7fded30eb93c] innobackupex(copy\_back()+0x377) [0x5a1767] innobackupex(main+0xacb) [0x591f9b] /lib64/libc.so.6(\_\_libc\_start\_main+0xfd) [0x7fded308ed5d] innobackupex() [0x585ea9] =============== server.conf # Ansible managed: /Users/hvo/Workspaces/Eclipse\_Helios/csw-cssm-engine/ha/deployment/ansible/roles/mariadb-prep/templates/mariadb\_server.cnf modified on 2016-01-03 21:37:29 by hvo on HVO-M-M09J # # These groups are read by MariaDB server. # Use it for options that only the server (but not clients) should see # # this is read by the standalone daemon and embedded servers [server] log-error=/var/log/mariadb.log # this is only for the mysqld standalone daemon [mysqld] socket=/var/lib/mysql/mysql.sock datadir=/var/lib/mysql # # \* Galera-related settings # [galera] binlog\_format=ROW default\_storage\_engine=innodb innodb\_autoinc\_lock\_mode=2 wsrep\_on=ON innodb\_doublewrite=1 wsrep\_provider=/usr/lib64/galera/libgalera\_smm.so wsrep\_provider\_options=gmcast.listen\_addr=tcp://0.0.0.0:4567; ist.recv\_addr=172.20.222.62:4568 # for IPv6 i need to insert a bracket around the address in the gcomm setting. look for it. wsrep\_cluster\_address=“gcomm://172.20.222.61,172.20.222.62” wsrep\_sst\_receive\_address=172.20.222.62:4444 wsrep\_node\_address=172.20.222.62:4567 wsrep\_cluster\_name=satelliteDatabaseCluster wsrep\_node\_name=dlo-lab62 wsrep\_sst\_method=xtrabackup-v2 wsrep\_sst\_auth=root:Test123 wsrep\_debug=ON bind-address=0.0.0.0 # this is only for embedded server [embedded] # This group is only read by MariaDB-5.5 servers. # If you use the same .cnf file for MariaDB of different versions, # use this group for options that older servers don’t understand [mysqld-10.1] # These two groups are only read by MariaDB servers, not by MySQL. # If you use the same .cnf file for MySQL and MariaDB, # you can put MariaDB-only options here [mariadb] query\_cache\_size=0 [mariadb-10.1] Thank you for your help, Huey ---
**Author:** ![hvo408](https://avatars.discourse-cdn.com/v4/letter/h/ecd19e/32.png) [@hvo408](https://forums.percona.com/u/hvo408)\ **Post date:** [January 4, 2016, 6:57pm UTC](https://forums.percona.com/t/innobackupex-does-not-seem-to-read-datadir-correctly/4643/2 "2016-01-04T18:57:18Z")
Ahh found the problem: permission of the config file server.cnf was wrong (600). Changed it to 644 and the Percora scripts can now read the config values fine. I wish the tool would spit out some errors somewhere saying it can’t read this file that file. That would be helpful I think.