Hi everyone;
We have a db structure running as a cluster in a four-node structure. Cluster is working and sync is working properly. sometimes nodes 1 and 3 shut down. When I run systemctl status mysql on 1.node when I control it, “Active: failed (Result: exit-code) since Mon 2022-09-26 11:16:14 +03;” looks like. By the way, the cluster is working and there is no problem. When starting the cluster, I also need to specify that I started it with bootstrap.
Cluster is running and sync is working but when I check mysql service node 1 cluster service seems to fail.
root@prd-mysql8-03:/# systemctl status mysql
● mysql.service - Percona XtraDB Cluster
Loaded: loaded (/lib/systemd/system/mysql.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Mon 2022-09-26 11:16:14 +03; 59min ago
Process: 2579 ExecStartPre=/usr/bin/mysql-systemd start-pre (code=exited, status=0/SUCCESS)
Process: 2631 ExecStartPre=/usr/bin/mysql-systemd check-grastate (code=exited, status=0/SUCCESS)
Process: 2660 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
Process: 2662 ExecStartPre=/bin/sh -c VAR=`bash /usr/bin/mysql-systemd galera-recovery`; [ $? -eq 0 ] && systemctl set-environment _WSREP_START_POSITION=$VAR || exit 1 (code=exited, status=0/SUCCESS)
Process: 2710 ExecStart=/usr/sbin/mysqld $_WSREP_START_POSITION (code=exited, status=1/FAILURE)
Process: 2713 ExecStopPost=/usr/bin/mysql-systemd stop-post (code=exited, status=0/SUCCESS)
Main PID: 2710 (code=exited, status=1/FAILURE)
Status: "Server startup in progress"
Sep 26 11:16:13 prd-mysql8-03.tokenpay.prd systemd[1]: Starting Percona XtraDB Cluster...
Sep 26 11:16:14 prd-mysql8-03.tokenpay.prd systemd[1]: mysql.service: Main process exited, code=exited, status=1/FAILURE
Sep 26 11:16:14 prd-mysql8-03.tokenpay.prd mysql-systemd[2713]: WARNING: mysql pid file /var/run/mysqld/mysqld.pid empty or not readable
Sep 26 11:16:14 prd-mysql8-03.tokenpay.prd mysql-systemd[2713]: WARNING: mysql may be already dead
Sep 26 11:16:14 prd-mysql8-03.tokenpay.prd systemd[1]: mysql.service: Failed with result 'exit-code'.
Sep 26 11:16:14 prd-mysql8-03.tokenpay.prd systemd[1]: Failed to start Percona XtraDB Cluster.
other nodes appear active
root@prd-mysql8-01:~# systemctl status mysql
● mysql.service - Percona XtraDB Cluster
Loaded: loaded (/lib/systemd/system/mysql.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2022-09-26 11:21:11 +03; 43min ago
Process: 949261 ExecStartPre=/usr/bin/mysql-systemd start-pre (code=exited, status=0/SUCCESS)
Process: 949317 ExecStartPre=/usr/bin/mysql-systemd check-grastate (code=exited, status=0/SUCCESS)
Process: 949346 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
Process: 949350 ExecStartPre=/bin/sh -c VAR=`bash /usr/bin/mysql-systemd galera-recovery`; [ $? -eq 0 ] && systemctl set-environment _WSREP_START_POSITION=$VAR || exit 1 (code=exited, status=0/SUCCESS)
Process: 950381 ExecStartPost=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
Process: 950383 ExecStartPost=/usr/bin/mysql-systemd start-post $MAINPID (code=exited, status=0/SUCCESS)
Main PID: 949452 (mysqld)
Status: "Server is operational"
Tasks: 50 (limit: 9405)
Memory: 495.3M
CGroup: /system.slice/mysql.service
└─949452 /usr/sbin/mysqld --wsrep_start_position=725a8f30-3348-11ed-918b-9794851ac393:208
Sep 26 11:20:34 prd-mysql8-01.tokenpay.prd systemd[1]: Starting Percona XtraDB Cluster...
Sep 26 11:21:06 prd-mysql8-01.tokenpay.prd systemd[1]: mysql.service: Got notification message from PID 950198, but reception only permitted for main PID 949452
Sep 26 11:21:07 prd-mysql8-01.tokenpay.prd systemd[1]: mysql.service: Got notification message from PID 950242, but reception only permitted for main PID 949452
Sep 26 11:21:07 prd-mysql8-01.tokenpay.prd systemd[1]: mysql.service: Got notification message from PID 950242, but reception only permitted for main PID 949452
Sep 26 11:21:08 prd-mysql8-01.tokenpay.prd systemd[1]: mysql.service: Got notification message from PID 950242, but reception only permitted for main PID 949452
Sep 26 11:21:11 prd-mysql8-01.tokenpay.prd systemd[1]: mysql.service: Got notification message from PID 950242, but reception only permitted for main PID 949452
Sep 26 11:21:11 prd-mysql8-01.tokenpay.prd mysql-systemd[950383]: SUCCESS!
Sep 26 11:21:11 prd-mysql8-01.tokenpay.prd systemd[1]: Started Percona XtraDB Cluster.
------
root@prd-mysql8-02:~# systemctl status mysql
● mysql.service - Percona XtraDB Cluster
Loaded: loaded (/lib/systemd/system/mysql.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2022-09-26 11:21:51 +03; 42min ago
Process: 951938 ExecStartPre=/usr/bin/mysql-systemd start-pre (code=exited, status=0/SUCCESS)
Process: 951993 ExecStartPre=/usr/bin/mysql-systemd check-grastate (code=exited, status=0/SUCCESS)
Process: 952022 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
Process: 952025 ExecStartPre=/bin/sh -c VAR=`bash /usr/bin/mysql-systemd galera-recovery`; [ $? -eq 0 ] && systemctl set-environment _WSREP_START_POSITION=$VAR || exit 1 (code=exited, status=0/SUCCESS)
Process: 952726 ExecStartPost=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
Process: 952730 ExecStartPost=/usr/bin/mysql-systemd start-post $MAINPID (code=exited, status=0/SUCCESS)
Main PID: 952122 (mysqld)
Status: "Server is operational"
Tasks: 57 (limit: 9405)
Memory: 433.4M
CGroup: /system.slice/mysql.service
└─952122 /usr/sbin/mysqld --wsrep_start_position=725a8f30-3348-11ed-918b-9794851ac393:208
Sep 26 11:21:42 prd-mysql8-02.tokenpay.prd systemd[1]: Starting Percona XtraDB Cluster...
Sep 26 11:21:51 prd-mysql8-02.tokenpay.prd mysql-systemd[952730]: SUCCESS!
Sep 26 11:21:51 prd-mysql8-02.tokenpay.prd systemd[1]: Started Percona XtraDB Cluster.
------
root@dr-mysql8-01:/var/run/mysqld# systemctl status mysql
● mysql.service - Percona XtraDB Cluster
Loaded: loaded (/lib/systemd/system/mysql.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2022-09-26 11:43:35 +03; 31min ago
Process: 2400 ExecStartPre=/usr/bin/mysql-systemd start-pre (code=exited, status=0/SUCCESS)
Process: 2451 ExecStartPre=/usr/bin/mysql-systemd check-grastate (code=exited, status=0/SUCCESS)
Process: 2480 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
Process: 2482 ExecStartPre=/bin/sh -c VAR=`bash /usr/bin/mysql-systemd galera-recovery`; [ $? -eq 0 ] && systemctl set-environment _WSREP_START_POSITION=$VAR || exit 1 (code=exited, status=0/SUCCESS)
Process: 4583 ExecStartPost=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
Process: 4585 ExecStartPost=/usr/bin/mysql-systemd start-post $MAINPID (code=exited, status=0/SUCCESS)
Main PID: 2581 (mysqld)
Status: "Server is operational"
Tasks: 130 (limit: 9405)
Memory: 617.5M
CGroup: /system.slice/mysql.service
└─2581 /usr/sbin/mysqld --wsrep_start_position=725a8f30-3348-11ed-918b-9794851ac393:214
Sep 26 11:38:54 dr-mysql8-01.tokenpay.dr systemd[1]: Starting Percona XtraDB Cluster...
Sep 26 11:43:32 dr-mysql8-01.tokenpay.dr systemd[1]: mysql.service: Got notification message from PID 4419, but reception only permitted for main PID 2581
Sep 26 11:43:32 dr-mysql8-01.tokenpay.dr systemd[1]: mysql.service: Got notification message from PID 4463, but reception only permitted for main PID 2581
Sep 26 11:43:33 dr-mysql8-01.tokenpay.dr systemd[1]: mysql.service: Got notification message from PID 4463, but reception only permitted for main PID 2581
Sep 26 11:43:33 dr-mysql8-01.tokenpay.dr systemd[1]: mysql.service: Got notification message from PID 4463, but reception only permitted for main PID 2581
Sep 26 11:43:34 dr-mysql8-01.tokenpay.dr systemd[1]: mysql.service: Got notification message from PID 4463, but reception only permitted for main PID 2581
Sep 26 11:43:35 dr-mysql8-01.tokenpay.dr mysql-systemd[4585]: SUCCESS!
Sep 26 11:43:35 dr-mysql8-01.tokenpay.dr systemd[1]: Started Percona XtraDB Cluster.
Additional info:
root@prd-mysql8-03:/# vi /data/mysql/grastate.dat
# GALERA saved state
version: 2.1
uuid: 725a8f30-3348-11ed-918b-9794851ac393
seqno: -1
safe_to_bootstrap: 0