# Xtrabackup first time replication failing on slave side

**URL:** <https://forums.percona.com/t/xtrabackup-first-time-replication-failing-on-slave-side/29345>\
**Category:** Percona XtraBackup\
**Created:** [March 26, 2024, 1:05am UTC](https://forums.percona.com/t/xtrabackup-first-time-replication-failing-on-slave-side/29345 "2024-03-26T01:05:18Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![CL69](https://avatars.discourse-cdn.com/v4/letter/c/c4cdca/32.png) [@CL69](https://forums.percona.com/u/CL69)\
**Post date:** [March 26, 2024, 1:05am UTC](https://forums.percona.com/t/xtrabackup-first-time-replication-failing-on-slave-side/29345/1 "2024-03-26T01:05:18Z")

</div>

Hi All,

I’ve followed numerous guides to set up a simple master slave replication on 2 Ubuntu servers running Mysql v8. I have v8 of Xtrabackup too:

_xtrabackup version 8.0.35-30 based on MySQL server 8.0.35 Linux (x86\_64) (revision id: 6beb4b49)_

The process I did was basically:

**ON MASTER:**

_sudo xtrabackup --backup --user=XYZ --password=ABC --target-dir=/backupfolderonmaster_

_sudo xtrabackup --prepare --target-dir=/backupfolderonmaster_

_rsync -avP /backupfolderonmaster/ USER@SLAVESERVERIP:/backupfolderonslave/_

**ON SLAVE:**

_sudo systemctl stop mysql_

_rm -rf /var/lib/mysql/_\*

_sudo xtrabackup --copy-back --target-dir=/backupfolderonslave_

**THIS IS MY FIRST PROBLEM ==\>**

_root@SLAVESERVER:~# sudo xtrabackup --copy-back --target-dir=/backupfolderonslave_  
\*2024-03-26T00:30:56.407087-00:00 0 [Note] [MY-011825] [Xtrabackup] recognized client arguments: --copy-back=1 --target-dir=/backupfolderonslave \*  
_xtrabackup version 8.0.35-30 based on MySQL server 8.0.35 Linux (x86\_64) (revision id: 6beb4b49)_  
_2024-03-26T00:30:56.407151-00:00 0 [ERROR] [MY-011825] [Xtrabackup] datadir must be specified._

So I try with:

_sudo xtrabackup --copy-back --target-dir=/backupfolderonslave/ --datadir=/var/lib/mysql/_

This copies files across and seems to be happy.

_2024-03-26T00:00:50.418982-00:00 0 [Note] [MY-011825] [Xtrabackup] completed OK!_

I then do this:

_chown -R mysql:mysql /var/lib/mysql_

**but then MY SECOND PROBLEM ==\>**

_root@SLAVESERVER:~# service mysql start_  
_Job for mysql.service failed because the control process exited with error code._  
_See “systemctl status mysql.service” and “journalctl -xeu mysql.service” for details._

I then run the following:

_root@SLAVESERVER:~# journalctl -xeu mysql.service_  
_░░ A start job for unit mysql.service has begun execution._  
\*░░ \*  
_░░ The job identifier is 45068._  
_Mar 26 00:50:24 SLAVESERVERIP mysql-systemd-start[2862328]: MySQL system database not found in /var/lib/mysql. Please run mysqld --initialize._  
_Mar 26 00:50:24 SLAVESERVERIP systemd[1]: mysql.service: Control process exited, code=exited, status=1/FAILURE_

Finally I run:

_root@SLAVESERVER:~# mysqld --initialize_  
_root@SLAVESERVER:~# service mysql start_

And MySQL starts on the SERVER but the server MySQL interface is asking me for my credentials.

“_**Warning! Webmin needs to know your MySQL administration login and password in order to manage your database. Please enter your administration username (usually root) and password below**_.”

Have I skipped a step or done something wrong? Just really confused why I’m being asked for MySQL details on the slave and what those details are. Is it something to do with the my.cnf file from the master?

Sorry for the potentially silly question but I have spent days trying to figure this out to no avail …

Thanks!

---

<div class="post-metadata">

**Author:** ![matthewb](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/matthewb/32/34_2.png) [@matthewb](https://forums.percona.com/u/matthewb)\
**Post date:** [March 26, 2024, 3:18pm UTC](https://forums.percona.com/t/xtrabackup-first-time-replication-failing-on-slave-side/29345/2 "2024-03-26T15:18:14Z")

</div>

> [@CL69](#):
>
> _root@SLAVESERVER:~# mysqld --initialize_

If you copied over a backup from source, you should not need to use this command.

Your `rsync` command could have copied it directly into the mysql datadir.

Did you copy the /etc/my.cnf from source over to replica?

Please look at mysql’s error log and not journalctl.

---

<div class="post-metadata">

**Author:** ![CL69](https://avatars.discourse-cdn.com/v4/letter/c/c4cdca/32.png) [@CL69](https://forums.percona.com/u/CL69)\
**Post date:** [March 27, 2024, 12:26am UTC](https://forums.percona.com/t/xtrabackup-first-time-replication-failing-on-slave-side/29345/3 "2024-03-27T00:26:22Z")

</div>

Thanks, I tried again from scratch and this time I made rsync send the files from the prepared Master backup to the Mysql datadir on the Slave (/var/lib/mysql) directly.

I also tried with copying the contents of /etc/mysql/my.cnf file from Master to Slave.

Still I have a problem in trying to start Mysql and when I look at the logs I see the suggestion of running " _mysqld --initialize_" on the Slave. When I look at Mysql logs I see nothing logged because I have stopped mysql before and no new messages are logged. I did however find this in syslog:

**mysql-systemd-start[3589017]: MySQL system database not found in /var/lib/mysql. Please run mysqld --initialize.**

I dont understand what could be going wrong? Is there another step I need to do? Is there a config file missing from Master to Slave? do I need to copy all the contents manually from the etc/mysql/ directory from master to slave too? … I’m lost at the moment so any more suggestions would be greatly appreciated.

---

<div class="post-metadata">

**Author:** ![matthewb](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/matthewb/32/34_2.png) [@matthewb](https://forums.percona.com/u/matthewb)\
**Post date:** [March 27, 2024, 4:04am UTC](https://forums.percona.com/t/xtrabackup-first-time-replication-failing-on-slave-side/29345/4 "2024-03-27T04:04:44Z")

</div>

> [@CL69](#):
>
> **mysql-systemd-start[3589017]: MySQL system database not found in /var/lib/mysql. Please run mysqld --initialize.**

Did you `chown -R mysql:mysql /var/lib/mysql` before you started? It’s very strange that you have data in /var/lib/mysql but “mysql” cannot see it. Please provide `ls -la /var/lib/mysql/`

---

<div class="post-metadata">

**Author:** ![CL69](https://avatars.discourse-cdn.com/v4/letter/c/c4cdca/32.png) [@CL69](https://forums.percona.com/u/CL69)\
**Post date:** [March 27, 2024, 2:47pm UTC](https://forums.percona.com/t/xtrabackup-first-time-replication-failing-on-slave-side/29345/5 "2024-03-27T14:47:18Z")

</div>

Thanks again - yes I did change the permissions on the SLAVE with:

chown -R mysql:mysql /var/lib/mysql

This is what i did exactly:

1. Creating a backup on the MASTER and storing it in a directory (/backupzpercona).
2. Preparing the backup on the MASTER with xtrabackup --prepare.
3. Stopping the MySQL service on the SLAVE.  
4)Transferring that backup using rsync to the /var/lib/mysql/ directory on the SLAVE.
4. changed directory ownership on SLAVE with chown -R mysql:mysql /var/lib/mysql
5. tried starting mysql server on SLAVE but that failed.

Here is the output of ls for the slave and master servers:

## **SLAVESERVER**

_ **root@SLAVESERVER:~# ls -la /var/lib/mysql/** _  
_total 1040488_  
_drwxr-x— 2 mysql mysql 4096 Mar 26 23:13 ‘#innodb\_redo’_  
_drwxrwxrwx 11 mysql mysql 4096 Mar 26 23:13 ._  
_drwxr-xr-x 70 root root 4096 Mar 26 23:13 .._  
_-rw-r----- 1 mysql mysql 447 Mar 26 23:04 backup-my.cnf_  
_drwxr-x— 2 mysql mysql 4096 Mar 26 23:04 dbthoruk_  
_drwxr-x— 2 mysql mysql 4096 Mar 26 23:04 dbthoruk\_b_  
_drwxr-x— 2 mysql mysql 4096 Mar 26 23:04 dbthoruk\_c_  
_drwxr-x— 2 mysql mysql 4096 Mar 26 23:04 dbthoruk\_r_  
_drwxr-x— 2 mysql mysql 4096 Mar 26 23:04 dbthoruk\_v_  
_-rw-r----- 1 mysql mysql 13222 Mar 26 23:04 ib\_buffer\_pool_  
_-rw-r----- 1 mysql mysql 79691776 Mar 26 23:13 ibdata1_  
_-rw-r----- 1 mysql mysql 12582912 Mar 26 23:13 ibtmp1_  
_-rw-r----- 1 mysql mysql 157 Mar 26 23:04 mysql-bin.000248_  
_-rw-r----- 1 mysql mysql 32 Mar 26 23:04 mysql-bin.index_  
_-rw-r----- 1 mysql mysql 33554432 Mar 26 23:13 mysql.ibd_  
_drwxr-x— 2 mysql mysql 4096 Mar 26 23:04 performance\_schema_  
_drwxr-x— 2 mysql mysql 12288 Mar 26 23:04 prod\_dbthoruk_  
_drwxr-x— 2 mysql mysql 4096 Mar 26 23:03 sys_  
_-rw-r----- 1 mysql mysql 452984832 Mar 26 23:13 undo\_001_  
_-rw-r----- 1 mysql mysql 452984832 Mar 26 23:13 undo\_002_  
_-rw-r----- 1 mysql mysql 21 Mar 26 23:04 xtrabackup\_binlog\_info_  
_-rw-r----- 1 mysql mysql 149 Mar 26 23:13 xtrabackup\_checkpoints_  
_-rw-r----- 1 mysql mysql 506 Mar 26 23:04 xtrabackup\_info_  
_-rw-r----- 1 mysql mysql 33554432 Mar 26 23:13 xtrabackup\_logfile_  
_-rw-r----- 1 mysql mysql 39 Mar 26 23:13 xtrabackup\_tablespaces_

## **MASTERSERVER (ls of prepared backup directory using xtrabackup)**

_ **root@MASTERSERVER:~# ls -la /backupzpercona** _  
_total 1040480_  
_drwxr-x— 2 root root 4096 Mar 26 23:13 ‘#innodb\_redo’_  
_drwxrwxrwx 11 root root 4096 Mar 26 23:13 ._  
_drwxr-xr-x 20 root root 4096 Mar 22 01:26 .._  
_-rw-r----- 1 root root 447 Mar 26 23:04 backup-my.cnf_  
_drwxr-x— 2 root root 4096 Mar 26 23:04 dbthoruk_  
_drwxr-x— 2 root root 4096 Mar 26 23:04 dbthoruk\_b_  
_drwxr-x— 2 root root 4096 Mar 26 23:04 dbthoruk\_c_  
_drwxr-x— 2 root root 4096 Mar 26 23:04 dbthoruk\_r_  
_drwxr-x— 2 root root 4096 Mar 26 23:04 dbthoruk\_v_  
_-rw-r----- 1 root root 13222 Mar 26 23:04 ib\_buffer\_pool_  
_-rw-r----- 1 root root 79691776 Mar 26 23:13 ibdata1_  
_-rw-r----- 1 root root 12582912 Mar 26 23:13 ibtmp1_  
_-rw-r----- 1 root root 157 Mar 26 23:04 mysql-bin.000248_  
_-rw-r----- 1 root root 32 Mar 26 23:04 mysql-bin.index_  
_-rw-r----- 1 root root 33554432 Mar 26 23:13 mysql.ibd_  
_drwxr-x— 2 root root 4096 Mar 26 23:04 performance\_schema_  
_drwxr-x— 2 root root 4096 Mar 26 23:04 prod\_dbthoruk_  
_drwxr-x— 2 root root 4096 Mar 26 23:03 sys_  
_-rw-r----- 1 root root 452984832 Mar 26 23:13 undo\_001_  
_-rw-r----- 1 root root 452984832 Mar 26 23:13 undo\_002_  
_-rw-r----- 1 root root 21 Mar 26 23:04 xtrabackup\_binlog\_info_  
_-rw-r----- 1 root root 149 Mar 26 23:13 xtrabackup\_checkpoints_  
_-rw-r----- 1 root root 506 Mar 26 23:04 xtrabackup\_info_  
_-rw-r----- 1 root root 33554432 Mar 26 23:13 xtrabackup\_logfile_  
_-rw-r----- 1 root root 39 Mar 26 23:13 xtrabackup\_tablespaces_

## **MASTERSERVER**

_ **root@MASTERSERVER:~# ls -la /var/lib/mysql/** _  
_total 1022052_  
_-rw-r----- 1 mysql mysql 196608 Mar 27 14:33 ‘#ib\_16384\_0.dblwr’_  
_-rw-r----- 1 mysql mysql 8585216 Mar 27 14:33 ‘#ib\_16384\_1.dblwr’_  
_drwxr-x— 2 mysql mysql 4096 Mar 27 14:33 ‘#innodb\_redo’_  
_drwxr-x— 2 mysql mysql 4096 Mar 18 01:33 ‘#innodb\_temp’_  
_drwx------ 12 mysql mysql 69632 Mar 21 01:50 ._  
_drwxr-xr-x 62 root root 4096 Feb 29 22:52 .._  
_-rw-r----- 1 mysql mysql 56 May 17 2022 auto.cnf_  
_-rw------- 1 mysql mysql 1676 May 17 2022 ca-key.pem_  
_-rw-r–r-- 1 mysql mysql 1112 May 17 2022 ca.pem_  
_-rw-r–r-- 1 mysql mysql 1112 May 17 2022 client-cert.pem_  
_-rw------- 1 mysql mysql 1680 May 17 2022 client-key.pem_  
_drwxr-x— 2 mysql mysql 12288 Nov 28 23:04 dbthoruk_  
_drwxr-x— 2 mysql mysql 4096 Jul 25 2023 dbthoruk\_b_  
_drwxr-x— 2 mysql mysql 4096 Aug 18 2023 dbthoruk\_c_  
_drwxr-x— 2 mysql mysql 4096 Aug 17 2023 dbthoruk\_r_  
_drwxr-x— 2 mysql mysql 4096 Aug 19 2023 dbthoruk\_v_  
_-rw-r–r-- 1 mysql mysql 0 Jun 1 2023 debian-5.7.flag_  
_-rw-r----- 1 mysql mysql 13222 Mar 16 15:11 ib\_buffer\_pool_  
_-rw-r----- 1 mysql mysql 79691776 Mar 27 14:33 ibdata1_  
_-rw-r----- 1 mysql mysql 12582912 Mar 18 01:23 ibtmp1_  
_-rw-r----- 1 mysql mysql 7 May 20 2022 localhost.pid_  
_-rw-r----- 1 mysql mysql 33554432 Mar 27 14:33 mysql.ibd_  
_-rw-r----- 1 mysql mysql 6 Jun 1 2023 mysql\_upgrade\_info_  
_-rw-r----- 1 mysql mysql 176 Mar 21 01:06 mysqld-auto.cnf_  
_drwxr-x— 2 mysql mysql 4096 Jun 1 2023 performance\_schema_  
_-rw------- 1 mysql mysql 1680 May 17 2022 private\_key.pem_  
_drwxr-x— 2 mysql mysql 12288 Mar 25 15:36 prod\_dbthoruk_  
_-rw-r–r-- 1 mysql mysql 452 May 17 2022 public\_key.pem_  
_-rw-r–r-- 1 mysql mysql 1112 May 17 2022 server-cert.pem_  
_-rw------- 1 mysql mysql 1676 May 17 2022 server-key.pem_  
_drwxr-x— 2 mysql mysql 4096 May 17 2022 sys_  
_-rw-r----- 1 mysql adm 5779781 Sep 4 2023 dbthoruk.log_  
_-rw-r----- 1 mysql mysql 7 Mar 18 01:23 dbthoruk.pid_  
_-rw-r----- 1 mysql mysql 452984832 Mar 27 14:33 undo\_001_  
_-rw-r----- 1 mysql mysql 452984832 Mar 27 14:33 undo\_002_

Another thought, could apparmour be affecting anything?

_root@SLAVESERVER:~# ls /etc/apparmor.d/ | grep -E ‘mysql|xtrabackup’_  
_usr.sbin.mysqld_

---

<div class="post-metadata">

**Author:** ![matthewb](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/matthewb/32/34_2.png) [@matthewb](https://forums.percona.com/u/matthewb)\
**Post date:** [March 27, 2024, 3:07pm UTC](https://forums.percona.com/t/xtrabackup-first-time-replication-failing-on-slave-side/29345/6 "2024-03-27T15:07:08Z")

</div>

> [@CL69](#):
>
> - 4)Transferring that backup using rsync to the /var/lib/mysql/ directory on the SLAVE.

You did not mention that you first erased the entire datadir _before_ doing the rsync. I want to verify that you did do that.

What was the output of your prepare step? Did it say “completed OK” at the end?

> apparmour

Yes, I would just disable it entirety. If mysql then starts, you know that’s the issue and you’ll need to find a guide on how to fix that.

---

<div class="post-metadata">

**Author:** ![CL69](https://avatars.discourse-cdn.com/v4/letter/c/c4cdca/32.png) [@CL69](https://forums.percona.com/u/CL69)\
**Post date:** [March 28, 2024, 4:22pm UTC](https://forums.percona.com/t/xtrabackup-first-time-replication-failing-on-slave-side/29345/7 "2024-03-28T16:22:48Z")

</div>

Thanks again for the suggestions. I had a nightmare last night but in the end it pointed me to a potential culprit here - the datadir of Mysql appears to have been corrupt somehow (perhaps through purging binary logs or something along those lines). I found this out the hard way when a reboot of the Master server would not allow Mysql to restart so the same problem as I was seeing on the slave.

I will now try this process again but fingers crossed should be better …