Hi percona community,
I have 3 nodes with percona xtraDB cluster installed on all of them.I am trying to configure the xtraDB cluster and when I run systemctl start mysql@bootstrap.service I get the following error:
Job for mysql@bootstrap.service failed because a timeout was exceeded.
See “systemctl status mysql@bootstrap.service” and “journalctl -xe” for details.
When I run systemctl status mysql@bootstrap.service this is what I get:
Feb 19 10:39:06 prod-mysql-node01.ipsl.co.ke systemd[1]: mysql@bootstrap.service: start operation timed out. Terminating.
Feb 19 10:40:36 prod-mysql-node01.ipsl.co.ke systemd[1]: mysql@bootstrap.service: State ‘stop-sigterm’ timed out. Killing.
Feb 19 10:40:36 prod-mysql-node01.ipsl.co.ke systemd[1]: mysql@bootstrap.service: Killing process 116410 (mysqld) with signal>
Feb 19 10:40:36 prod-mysql-node01.ipsl.co.ke systemd[1]: mysql@bootstrap.service: Main process exited, code=killed, status=9/>
Feb 19 10:40:36 prod-mysql-node01.ipsl.co.ke mysql-systemd[116552]: /usr/bin/mysql-systemd: line 233: kill: (116410) - No suc>
Feb 19 10:40:36 prod-mysql-node01.ipsl.co.ke mysql-systemd[116552]: WARNING: mysql already dead
Feb 19 10:40:36 prod-mysql-node01.ipsl.co.ke mysql-systemd[116552]: ERROR! Stale PID file: /var/run/mysqld/mysqld.pid
Feb 19 10:40:36 prod-mysql-node01.ipsl.co.ke systemd[1]: mysql@bootstrap.service: Control process exited, code=exited status=3
Feb 19 10:40:36 prod-mysql-node01.ipsl.co.ke systemd[1]: mysql@bootstrap.service: Failed with result ‘timeout’.
Feb 19 10:40:36 prod-mysql-node01.ipsl.co.ke systemd[1]: Failed to start Percona XtraDB Cluster with config /etc/sysconfig/my>
lines 1-21/21 (END)
This is my log file:
[root@prod-mysql-node01 log]# tail -f mysqld.log
2024-02-19T07:37:37.496153Z 0 [Note] [MY-000000] [Galera] Server initialized
2024-02-19T07:37:37.496163Z 0 [Note] [MY-000000] [WSREP] Server status change initializing → initialized
2024-02-19T07:37:37.496180Z 0 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2024-02-19T07:37:37.496236Z 2 [Note] [MY-000000] [Galera] Bootstrapping a new cluster, setting initial position to 00000000-0000-0000-0000-000000000000:-1
2024-02-19T07:37:37.497612Z 8 [Warning] [MY-013185] [Server] Currently unknown variable ‘clone_valid_donor_list’ was read from the persisted config file.
2024-02-19T07:37:37.497671Z 8 [Note] [MY-000000] [Galera] pause
2024-02-19T07:37:37.500525Z 7 [Note] [MY-000000] [WSREP] Cluster table is empty, not recovering transactions
2024-02-19T07:37:37.500570Z 2 [Note] [MY-000000] [WSREP] Server status change initialized → joined
2024-02-19T07:37:37.500582Z 2 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2024-02-19T07:37:37.500596Z 2 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
What could I be missing?I am running it on rocky linux 8.9
Hi matthewb,
After I removed the parameter,the bootstrap did start, however when I try to join in the 2nd node by starting mysql,I get the following error:
You need to copy the SSL certificates from node1 over to node2 before starting node2. Or, more simply, disable pxc_encrypt_cluster_traffic on both nodes.
Hi matthewb,
I am now getting this error when I try to bootstrap:
[root@prod-mysql-node01 log]# tail -f mysqld.log
ERROR! WSREP: Failed to recover position:
Log of wsrep recovery (–wsrep-recover):
INFO: WSREP: Running position recovery with --log_error=‘/data01/mysql_data/mysql/wsrep_recovery_verbose.E1jd1l’ --pid-file=‘/data01/mysql_data/mysql/prod-mysql-node01.ipsl.co.ke-recover.pid’
ERROR! WSREP: Failed to recover position:
Log of wsrep recovery (–wsrep-recover):
INFO: WSREP: Running position recovery with --log_error=‘/data01/mysql_data/mysql/wsrep_recovery_verbose.xKt2xs’ --pid-file=‘/data01/mysql_data/mysql/prod-mysql-node01.ipsl.co.ke-recover.pid’
ERROR! WSREP: Failed to recover position:
Log of wsrep recovery (–wsrep-recover):
INFO: WSREP: Running position recovery with --log_error=‘/data01/mysql_data/mysql/wsrep_recovery_verbose.BufNcm’ --pid-file=‘/data01/mysql_data/mysql/prod-mysql-node01.ipsl.co.ke-recover.pid’
ERROR! WSREP: Failed to recover position:
Please attach more logs. Those last repeated lines of the same content don’t really help much. Is this on node1 after you disabled the cluster traffic encryption? Please provide a more detailed list of actions/steps/commands taken.
Hi matthewb,
This is after I removed the parameter.I am now trying to bootstrap the 1st node using the bootstrap command: systemctl start mysql@bootstrap.service
And its not giving any other logs apart from the one I shared initially.
@Cris Please stop mysql. Ensure all mysqld instances are not running (verify with ps -Af | grep mysqld). Then zero out the mysql error log (ex: echo >/path/to/file.log). Then try bootstrap. Then, attach the entire log file.
Hi matthewb,
I did another clean upgrade from MySQL 8.0.32 to Percona MySQL 8.0.32 and ran the bootstrap command which started well and its currently active. Now on trying to join the 2nd node, I am getting the error as attached on the file named node 2.Its not that descriptive so kind of difficult to troubleshoot.
I see where node2 failed, but the reason is in node1’s log, which you provided an unaligned timestamp picture. Please find the same timestamps in nod1 log and look for errors.
Hi matthewb,
Attached are the two logfiles in a text file from where the run started on the 2 nodes Node 2 logfile.txt (15.1 KB) Node 1 logfile.txt (12.1 KB)
So node1 is online, bootstrapped, and in PRIMARY state. But when you try to start node2, it will not join. 9/10 times this is network related. Read over your my.cnf again and check everything lines up. Provide both here if you wish.
