Failed to Start MySQL After Restoration

MySQL & MariaDB Percona XtraBackup
1

full backup：
xtrabackup --defaults-file=/etc/my.cnf --user=backupuser --password=xxxxx --socket=/var/lib/mysql/mysql.sock --compress-threads=4 --stream=xbstream --compress --backup --no-version-check=1 > /root/backup/backup1.qp

prepare

xbstream -x -p 4 -C /root/data < /root/backup/backup1.qp
xtrabackup --decompress --target-dir=/root/data --parallel=4
xtrabackup --prepare --target-dir=/root/data

then I stop mysql and remove the data file

service mysqld stop
mv /var/lib/mysql /var/lib/mysql2

Restoring:
xtrabackup --defaults-file=/etc/my.cnf --copy-back --target-dir=/root/data

change the File Owner and Group to mysql
chown -R mysql:mysql /var/lib/mysql

check:

-rw-r-----. 1 mysql mysql      157 Nov 10 16:21 binlog.000004
-rw-r-----. 1 mysql mysql       14 Nov 10 16:21 binlog.index
drwxr-x---. 2 mysql mysql       60 Nov 10 16:21 DB1
drwxr-x---. 2 mysql mysql       60 Nov 10 16:21 DB2
-rw-r-----. 1 mysql mysql     3542 Nov 10 16:21 ib_buffer_pool
-rw-r-----. 1 mysql mysql 12582912 Nov 10 16:21 ibdata1
-rw-r-----. 1 mysql mysql 50331648 Nov 10 16:21 ib_logfile0
-rw-r-----. 1 mysql mysql 50331648 Nov 10 16:21 ib_logfile1
-rw-r-----. 1 mysql mysql 12582912 Nov 10 16:21 ibtmp1
drwxr-x---. 2 mysql mysql      143 Nov 10 16:21 mysql
-rw-r-----. 1 mysql mysql 25165824 Nov 10 16:21 mysql.ibd
drwxr-x---. 2 mysql mysql     8192 Nov 10 16:21 performance_schema
drwxr-x---. 2 mysql mysql       28 Nov 10 16:21 sys
-rw-r-----. 1 mysql mysql 16777216 Nov 10 16:21 undo_001
-rw-r-----. 1 mysql mysql 16777216 Nov 10 16:21 undo_002
-rw-r-----. 1 mysql mysql      605 Nov 10 16:21 xtrabackup_info
-rw-r-----. 1 mysql mysql        1 Nov 10 16:21 xtrabackup_master_key_id

then i start mysql. find error.
command: service mysqld start
Error message :

Redirecting to /bin/systemctl start mysqld.service
Job for mysqld.service failed because the control process exited with error code. See "systemctl status mysqld.service" and "journalctl -xe" for details.

see mysql log

mysqld: File './binlog.index' not found (OS errno 13 - Permission denied)
2022-11-10T08:21:43.881556Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.28) starting as process 15585
2022-11-10T08:21:43.882576Z 0 [Warning] [MY-013242] [Server] --character-set-server: 'utf8' is currently an alias for the character set UTF8MB3, but will be an alias for UTF8MB4 in a future release. Please consider using UTF8MB4 in order to be unambiguous.
2022-11-10T08:21:43.885546Z 0 [ERROR] [MY-010119] [Server] Aborting
2022-11-10T08:21:43.885803Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.28)  MySQL Community Server - GPL.

I can’t understand. because in /var/lib/mysql ，All file permissions are OK.

when after delete /var/lib/mysql and move mysql2 to mysql, mysql start sucess.

rm -rf /var/lib/mysql
mv /var/lib/mysql2 /var/lib/mysql

mysql is “mysql-8.0.28-1.el7.x86_64”, and xtrabackup is “percona-xtrabackup-80-8.0.28-21.1.el7.x86_64”
OS is centOS7.6

anyone can help me?

1 Like
2

Hi @chen3438716

Can you try it with SELinux disabled?

setenforce 0

If it doesn’t works, what happens if you run mysqld from command line?

mysqld --defaults-file=/etc/my.cnf

Please send us the full error log.

1 Like
3

I don’t know why must set SELinux disabled? Can we keep SELinux the same?

[root@host-8-40-147-157 lib]# mysqld --defaults-file=/etc/my.cnf
2022-11-10T11:26:00.211761Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.28) starting as process 25414
2022-11-10T11:26:00.216771Z 0 [Warning] [MY-013242] [Server] --character-set-server: 'utf8' is currently an alias for the character set UTF8MB3, but will be an alias for UTF8MB4 in a future release. Please consider using UTF8MB4 in order to be unambiguous.
2022-11-10T11:26:00.217450Z 0 [ERROR] [MY-010123] [Server] Fatal error: Please read "Security" section of the manual to find out how to run mysqld as root!
2022-11-10T11:26:00.217523Z 0 [ERROR] [MY-010119] [Server] Aborting
2022-11-10T11:26:00.217770Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.28)  MySQL Community Server - GPL.
1 Like
4

I don’t know why must set SELinux disabled? Can we keep SELinux the same?

The intend is to test if SELinux is blocking mysqld from reading the file as you are having a permission issue.

2022-11-10T11:26:00.217450Z 0 [ERROR] [MY-010123] [Server] Fatal error: Please read "Security" section of the manual to find out how to run mysqld as root!

Please run mysqld with --user=mysql:

mysqld --defaults-file=/etc/my.cnf --user=mysql
1 Like
5

when set SELinux disabled, mysql start success.

when i run: mysqld --defaults-file=/etc/my.cnf --user=mysql

it is in pending

the /var/log/mysqld.log：

2022-11-10T13:26:38.637001Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.28) starting as process 4189
2022-11-10T13:26:38.638036Z 0 [Warning] [MY-013242] [Server] --character-set-server: 'utf8' is currently an alias for the character set UTF8MB3, but will be an alias for UTF8MB4 in a future release. Please consider using UTF8MB4 in order to be unambiguous.
2022-11-10T13:26:38.643029Z 0 [Warning] [MY-010075] [Server] No existing UUID has been found, so we assume that this is the first time that this server has been started. Generating a new UUID: 4e6195aa-60fb-11ed-9074-005056a8c495.
2022-11-10T13:26:38.660875Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2022-11-10T13:26:39.006953Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2022-11-10T13:26:39.156275Z 0 [System] [MY-010229] [Server] Starting XA crash recovery...
2022-11-10T13:26:39.164802Z 0 [System] [MY-010232] [Server] XA crash recovery finished.
2022-11-10T13:26:39.499137Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
2022-11-10T13:26:39.499192Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel.
2022-11-10T13:26:39.675547Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: '::' port: 33060, socket: /var/run/mysqld/mysqlx.sock
2022-11-10T13:26:39.675599Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.28'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MySQL Community Server - GPL.
1 Like
6

Hello @chen3438716,
I’ve ran into this exact problem with SELinux. The problem is that you first extracted and prepared the backup inside /root/data which is not a recognized path by SELinux for mysql data. Thus when you ran --copy-back, the files were copied with the wrong selinux context. Had you extracted and prepared directly inside /var/lib/mysql, the correct context would have been applied and SELinux would not be blocking.

Going forward, stop mysql, then restore the context of all your mysql files:

restorecon -Rv /var/lib/mysql

You can run ls -Z before and after the above command to verify the contexts were changed. After that, put SELinux back into enforcing mode and start mysql. Look at the SELinux logs in /var/logs/audit if you keep getting issues when back in Enforcing mode. Switch to Permissive to allow mysql to start and still get the log entries.

2 Likes
7

Thanks very much.

using “restorecon -Rv /var/lib/mysql” , it is OK too

1 Like