# Can't restart one node

**URL:** <https://forums.percona.com/t/cant-restart-one-node/5001>\
**Category:** Percona XtraDB Cluster 5.x\
**Created:** [July 27, 2016, 5:46am UTC](https://forums.percona.com/t/cant-restart-one-node/5001 "2016-07-27T05:46:12Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![miracula](https://avatars.discourse-cdn.com/v4/letter/m/ec9cab/32.png) [@miracula](https://forums.percona.com/u/miracula)\
**Post date:** [July 27, 2016, 5:46am UTC](https://forums.percona.com/t/cant-restart-one-node/5001/1 "2016-07-27T05:46:12Z")

</div>

Hi everyone, I’m nervous as can be. This cluster has been working fine for a year, after I somehow reset it to position zero and lost everything. (Thus the nervousness).

I have one server (in a cluster of 3) that is showing disk errors, so I took one down for needed maintenance in advance of doing the disk swaps.

Now it won’t come back.

I swear nothing has changed, but as this system is incredibly reliable 'til it’s not, I can’t guarantee.

The lines that are confusing me in the error log are:  
Connecting to MySQL server host: localhost, user: (null), password: not set, port: 0, socket: /var/lib/mysql/mysql.sock  
-innobackupex-backup: Using server version 5.6.28-76.1-56-log  
-innobackupex-backup: innobackupex version 2.3.4 based on MySQL server 5.6.24 Linux (x86\_64) (revision id: e80c779)  
-innobackupex-backup: xtrabackup: uses posix\_fadvise().  
-innobackupex-backup: xtrabackup: cd to /var/lib/mysql  
-innobackupex-backup: xtrabackup: open files limit requested 0, set to 5000  
-innobackupex-backup: xtrabackup: using the following InnoDB configuration:  
-innobackupex-backup: xtrabackup: innodb\_data\_home\_dir = ./  
-innobackupex-backup: xtrabackup: innodb\_data\_file\_path = ibdata1:12M:autoextend  
-innobackupex-backup: xtrabackup: innodb\_log\_group\_home\_dir = ./  
-innobackupex-backup: xtrabackup: innodb\_log\_files\_in\_group = 2  
-innobackupex-backup: xtrabackup: innodb\_log\_file\_size = 20971520  
-innobackupex-backup: xtrabackup: using O\_DIRECT  
-innobackupex-backup: 160727 07:01:40 \>\> log scanned up to (70923364024)  
-innobackupex-backup: xtrabackup: Generating a list of tablespaces  
-innobackupex-backup: 2016-07-27 07:01:40 7f6534131740 InnoDB: Operating system error number 13 in a file operation.  
-innobackupex-backup: InnoDB: The error means mysqld does not have the access rights to  
-innobackupex-backup: InnoDB: the directory.  
-wsrep-sst-donor: innobackupex finished with error: 1. Check /var/lib/mysql//innobackup.backup.log  
-wsrep-sst-donor: Cleanup after exit with status:22

If I read this correctly, it’s trying to log in with no username or password, when I have it clearly listed in the /etc/my.cnf  
wsrep\_sst\_auth = x:x

It also says it can’t access anything in /var/lib/mysql, but that directory is owned by mysql:mysql. I even changed it to 777 to no avail.

There is no /var/lib/mysql/innobackup.backup.log - so something is definitely weird. The double slash part also seems odd - there is no trailing slash in /etc/my.cnf

Please point me in the right direction. My pants are down, 1 of 3 in the cluster is down, and one of the remaining 2 has a wonky disk. I don’t want another disaster like last year!

---

<div class="post-metadata">

**Author:** ![miracula](https://avatars.discourse-cdn.com/v4/letter/m/ec9cab/32.png) [@miracula](https://forums.percona.com/u/miracula)\
**Post date:** [August 4, 2016, 7:55am UTC](https://forums.percona.com/t/cant-restart-one-node/5001/2 "2016-08-04T07:55:40Z")

</div>

So, I figured it out. (Thanks all for you great ideas and help :p)

It turns out my backup guy had added a script on server that was to have been the donor, which left backup files in the /var/lib/mysql directory. They were owned by the backup process, not mysql, so the donor could not read those files, and errored out.

I took the cluster down to one node, started up the misbehaving box successfully, then re-added the box with bad backup files.
