# Unable to bootstrap the 1st node **URL:** \ **Category:** Uncategorized\ **Created:** [February 19, 2024, 7:53am UTC](https://forums.percona.com/t/unable-to-bootstrap-the-1st-node/28551 "2024-02-19T07:53:37Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![Cris](https://avatars.discourse-cdn.com/v4/letter/c/71e660/32.png) [@Cris](https://forums.percona.com/u/Cris)\ **Post date:** [February 19, 2024, 7:53am UTC](https://forums.percona.com/t/unable-to-bootstrap-the-1st-node/28551/1 "2024-02-19T07:53:37Z")
Hi percona community, I have 3 nodes with percona xtraDB cluster installed on all of them.I am trying to configure the xtraDB cluster and when I run systemctl start mysql@bootstrap.service I get the following error: Job for mysql@bootstrap.service failed because a timeout was exceeded. See “systemctl status mysql@bootstrap.service” and “journalctl -xe” for details. When I run systemctl status mysql@bootstrap.service this is what I get: [mysqluser@prod-mysql-node01 ~]$ systemctl status mysql@bootstrap.service ● mysql@bootstrap.service - Percona XtraDB Cluster with config /etc/sysconfig/mysql.bootstrap Loaded: loaded (/usr/lib/systemd/system/mysql@.service; disabled; vendor preset: disabled) Active: failed (Result: timeout) since Mon 2024-02-19 10:40:36 EAT; 29s ago Process: 116552 ExecStopPost=/usr/bin/mysql-systemd stop-post (code=exited, status=3) Process: 116410 ExecStart=/usr/sbin/mysqld $EXTRA\_ARGS $\_WSREP\_START\_POSITION (code=killed, signal=KILL) Process: 116304 ExecStartPre=/bin/sh -c VAR=`bash /usr/bin/mysql-systemd galera-recovery`; [$? -eq 0] && systemctl set-en\> Process: 116302 ExecStartPre=/bin/sh -c systemctl unset-environment \_WSREP\_START\_POSITION (code=exited, status=0/SUCCESS) Process: 116260 ExecStartPre=/usr/bin/mysql-systemd start-pre (code=exited, status=0/SUCCESS) Main PID: 116410 (code=killed, signal=KILL) Status: “Server startup in progress” Feb 19 10:39:06 prod-mysql-node01.ipsl.co.ke systemd[1]: mysql@bootstrap.service: start operation timed out. Terminating. Feb 19 10:40:36 prod-mysql-node01.ipsl.co.ke systemd[1]: mysql@bootstrap.service: State ‘stop-sigterm’ timed out. Killing. Feb 19 10:40:36 prod-mysql-node01.ipsl.co.ke systemd[1]: mysql@bootstrap.service: Killing process 116410 (mysqld) with signal\> Feb 19 10:40:36 prod-mysql-node01.ipsl.co.ke systemd[1]: mysql@bootstrap.service: Main process exited, code=killed, status=9/\> Feb 19 10:40:36 prod-mysql-node01.ipsl.co.ke mysql-systemd[116552]: /usr/bin/mysql-systemd: line 233: kill: (116410) - No suc\> Feb 19 10:40:36 prod-mysql-node01.ipsl.co.ke mysql-systemd[116552]: WARNING: mysql already dead Feb 19 10:40:36 prod-mysql-node01.ipsl.co.ke mysql-systemd[116552]: ERROR! Stale PID file: /var/run/mysqld/mysqld.pid Feb 19 10:40:36 prod-mysql-node01.ipsl.co.ke systemd[1]: mysql@bootstrap.service: Control process exited, code=exited status=3 Feb 19 10:40:36 prod-mysql-node01.ipsl.co.ke systemd[1]: mysql@bootstrap.service: Failed with result ‘timeout’. Feb 19 10:40:36 prod-mysql-node01.ipsl.co.ke systemd[1]: Failed to start Percona XtraDB Cluster with config /etc/sysconfig/my\> lines 1-21/21 (END) This is my log file: [root@prod-mysql-node01 log]# tail -f mysqld.log 2024-02-19T07:37:37.496153Z 0 [Note] [MY-000000] [Galera] Server initialized 2024-02-19T07:37:37.496163Z 0 [Note] [MY-000000] [WSREP] Server status change initializing → initialized 2024-02-19T07:37:37.496180Z 0 [Note] [MY-000000] [WSREP] wsrep\_notify\_cmd is not defined, skipping notification. 2024-02-19T07:37:37.496236Z 2 [Note] [MY-000000] [Galera] Bootstrapping a new cluster, setting initial position to 00000000-0000-0000-0000-000000000000:-1 2024-02-19T07:37:37.497612Z 8 [Warning] [MY-013185] [Server] Currently unknown variable ‘clone\_valid\_donor\_list’ was read from the persisted config file. 2024-02-19T07:37:37.497671Z 8 [Note] [MY-000000] [Galera] pause 2024-02-19T07:37:37.500525Z 7 [Note] [MY-000000] [WSREP] Cluster table is empty, not recovering transactions 2024-02-19T07:37:37.500570Z 2 [Note] [MY-000000] [WSREP] Server status change initialized → joined 2024-02-19T07:37:37.500582Z 2 [Note] [MY-000000] [WSREP] wsrep\_notify\_cmd is not defined, skipping notification. 2024-02-19T07:37:37.500596Z 2 [Note] [MY-000000] [WSREP] wsrep\_notify\_cmd is not defined, skipping notification. What could I be missing?I am running it on rocky linux 8.9 ---
**Author:** ![Ivan\_Groenewold](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/ivan_groenewold/32/6299_2.png) [@Ivan\_Groenewold](https://forums.percona.com/u/Ivan_Groenewold)\ **Post date:** [February 19, 2024, 1:37pm UTC](https://forums.percona.com/t/unable-to-bootstrap-the-1st-node/28551/2 "2024-02-19T13:37:33Z")
Hi, have you changed the location of the pidfile? it seems systemd is timing out waiting for the service to be up. ---
**Author:** ![Cris](https://avatars.discourse-cdn.com/v4/letter/c/71e660/32.png) [@Cris](https://forums.percona.com/u/Cris)\ **Post date:** [February 20, 2024, 5:47am UTC](https://forums.percona.com/t/unable-to-bootstrap-the-1st-node/28551/3 "2024-02-20T05:47:53Z")
Hi Ivan\_Groenewold, I have not changed it. My default location is /var/run/mysqld/mysqld.pid. ---
**Author:** ![matthewb](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/matthewb/32/34_2.png) [@matthewb](https://forums.percona.com/u/matthewb)\ **Post date:** [February 20, 2024, 5:01pm UTC](https://forums.percona.com/t/unable-to-bootstrap-the-1st-node/28551/4 "2024-02-20T17:01:35Z")
> [@Cris](#): > > Currently unknown variable ‘clone\_valid\_donor\_list’ was read from the persisted config file. Please remove this unknown parameter from your my.cnf and also from $datadir/mysqld-auto.cnf ---
**Author:** ![Cris](https://avatars.discourse-cdn.com/v4/letter/c/71e660/32.png) [@Cris](https://forums.percona.com/u/Cris)\ **Post date:** [February 21, 2024, 7:53am UTC](https://forums.percona.com/t/unable-to-bootstrap-the-1st-node/28551/5 "2024-02-21T07:53:30Z")
Hi matthewb, After I removed the parameter,the bootstrap did start, however when I try to join in the 2nd node by starting mysql,I get the following error: [root@prod-mysql-node03 log]# tail -f mysqld.log 2024-02-21T06:19:23.502248Z 0 [Warning] [MY-000000] [Galera] Handshake failed: tlsv1 alert decrypt error 2024-02-21T06:19:25.001472Z 0 [Warning] [MY-000000] [Galera] Handshake failed: tlsv1 alert decrypt error 2024-02-21T06:19:26.504612Z 0 [Warning] [MY-000000] [Galera] Handshake failed: tlsv1 alert decrypt error 2024-02-21T06:19:28.003667Z 0 [Warning] [MY-000000] [Galera] Handshake failed: tlsv1 alert decrypt error 2024-02-21T06:19:29.502814Z 0 [Warning] [MY-000000] [Galera] Handshake failed: tlsv1 alert decrypt error 2024-02-21T06:19:31.004831Z 0 [Warning] [MY-000000] [Galera] Handshake failed: tlsv1 alert decrypt error 2024-02-21T06:24:38.571926Z 0 [Warning] [MY-000000] [Galera] Handshake failed: tlsv1 alert decrypt error 2024-02-21T06:24:40.571613Z 0 [Warning] [MY-000000] [Galera] Handshake failed: tlsv1 alert decrypt error 2024-02-21T07:11:32.950841Z 0 [Warning] [MY-000000] [Galera] Handshake failed: tlsv1 alert decrypt error 2024-02-21T07:11:34.943237Z 0 [Warning] [MY-000000] [Galera] Handshake failed: tlsv1 alert decrypt error What could be the issue here? ---
**Author:** ![matthewb](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/matthewb/32/34_2.png) [@matthewb](https://forums.percona.com/u/matthewb)\ **Post date:** [February 21, 2024, 4:07pm UTC](https://forums.percona.com/t/unable-to-bootstrap-the-1st-node/28551/6 "2024-02-21T16:07:12Z")
You need to copy the SSL certificates from node1 over to node2 before starting node2. Or, more simply, disable `pxc_encrypt_cluster_traffic` on both nodes. ---
**Author:** ![Cris](https://avatars.discourse-cdn.com/v4/letter/c/71e660/32.png) [@Cris](https://forums.percona.com/u/Cris)\ **Post date:** [February 22, 2024, 11:17am UTC](https://forums.percona.com/t/unable-to-bootstrap-the-1st-node/28551/7 "2024-02-22T11:17:03Z")
Hi matthewb, I am now getting this error when I try to bootstrap: [root@prod-mysql-node01 log]# tail -f mysqld.log ERROR! WSREP: Failed to recover position: Log of wsrep recovery (–wsrep-recover): INFO: WSREP: Running position recovery with --log\_error=‘/data01/mysql\_data/mysql/wsrep\_recovery\_verbose.E1jd1l’ --pid-file=‘/data01/mysql\_data/mysql/prod-mysql-node01.ipsl.co.ke-recover.pid’ ERROR! WSREP: Failed to recover position: Log of wsrep recovery (–wsrep-recover): INFO: WSREP: Running position recovery with --log\_error=‘/data01/mysql\_data/mysql/wsrep\_recovery\_verbose.xKt2xs’ --pid-file=‘/data01/mysql\_data/mysql/prod-mysql-node01.ipsl.co.ke-recover.pid’ ERROR! WSREP: Failed to recover position: Log of wsrep recovery (–wsrep-recover): INFO: WSREP: Running position recovery with --log\_error=‘/data01/mysql\_data/mysql/wsrep\_recovery\_verbose.BufNcm’ --pid-file=‘/data01/mysql\_data/mysql/prod-mysql-node01.ipsl.co.ke-recover.pid’ ERROR! WSREP: Failed to recover position: ---
**Author:** ![matthewb](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/matthewb/32/34_2.png) [@matthewb](https://forums.percona.com/u/matthewb)\ **Post date:** [February 22, 2024, 4:03pm UTC](https://forums.percona.com/t/unable-to-bootstrap-the-1st-node/28551/8 "2024-02-22T16:03:14Z")
Please attach more logs. Those last repeated lines of the same content don’t really help much. Is this on node1 after you disabled the cluster traffic encryption? Please provide a more detailed list of actions/steps/commands taken. ---
**Author:** ![Cris](https://avatars.discourse-cdn.com/v4/letter/c/71e660/32.png) [@Cris](https://forums.percona.com/u/Cris)\ **Post date:** [February 23, 2024, 7:58am UTC](https://forums.percona.com/t/unable-to-bootstrap-the-1st-node/28551/9 "2024-02-23T07:58:31Z")
Hi matthewb, This is after I removed the parameter.I am now trying to bootstrap the 1st node using the bootstrap command: systemctl start mysql@bootstrap.service And its not giving any other logs apart from the one I shared initially. ---
**Author:** ![matthewb](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/matthewb/32/34_2.png) [@matthewb](https://forums.percona.com/u/matthewb)\ **Post date:** [February 26, 2024, 5:35pm UTC](https://forums.percona.com/t/unable-to-bootstrap-the-1st-node/28551/10 "2024-02-26T17:35:34Z")
@Cris Please stop mysql. Ensure all mysqld instances are not running (verify with ps -Af | grep mysqld). Then zero out the mysql error log (ex: echo \>/path/to/file.log). Then try bootstrap. Then, attach the entire log file. ---
**Author:** ![Cris](https://avatars.discourse-cdn.com/v4/letter/c/71e660/32.png) [@Cris](https://forums.percona.com/u/Cris)\ **Post date:** [February 27, 2024, 10:49am UTC](https://forums.percona.com/t/unable-to-bootstrap-the-1st-node/28551/11 "2024-02-27T10:49:14Z")
Hi matthewb, I did another clean upgrade from MySQL 8.0.32 to Percona MySQL 8.0.32 and ran the bootstrap command which started well and its currently active. Now on trying to join the 2nd node, I am getting the error as attached on the file named node 2.Its not that descriptive so kind of difficult to troubleshoot. ![Node 1](https://us1.discourse-cdn.com/flex019/uploads/percona1/original/2X/1/1de35ebaf77706564ac1ba75e554630bdc539f42.png) ![Node 2](https://us1.discourse-cdn.com/flex019/uploads/percona1/original/2X/d/d853edda31eb51ab66de44b187158f87ae698bfd.png) See if you can assist. I have also attached part of node 1 logfile ---
**Author:** ![matthewb](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/matthewb/32/34_2.png) [@matthewb](https://forums.percona.com/u/matthewb)\ **Post date:** [February 27, 2024, 3:03pm UTC](https://forums.percona.com/t/unable-to-bootstrap-the-1st-node/28551/12 "2024-02-27T15:03:45Z")
I see where node2 failed, but the reason is in node1’s log, which you provided an unaligned timestamp picture. Please find the same timestamps in nod1 log and look for errors. ---
**Author:** ![Cris](https://avatars.discourse-cdn.com/v4/letter/c/71e660/32.png) [@Cris](https://forums.percona.com/u/Cris)\ **Post date:** [February 28, 2024, 5:40am UTC](https://forums.percona.com/t/unable-to-bootstrap-the-1st-node/28551/13 "2024-02-28T05:40:07Z")
Hi matthewb, Attached are the two logfiles in a text file from where the run started on the 2 nodes [Node 2 logfile.txt](https://forums.percona.com/uploads/short-url/73m0aP84EehrJIsfQ8m1yKyxXLo.txt) (15.1 KB) [Node 1 logfile.txt](https://forums.percona.com/uploads/short-url/rG8TjqZAd14yXg08FIONb189k4O.txt) (12.1 KB) ---
**Author:** ![matthewb](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/matthewb/32/34_2.png) [@matthewb](https://forums.percona.com/u/matthewb)\ **Post date:** [February 29, 2024, 10:06pm UTC](https://forums.percona.com/t/unable-to-bootstrap-the-1st-node/28551/14 "2024-02-29T22:06:42Z")
Both logs show ‘operation canceled’. Make sure ports 4444, 4567, and 4568 are open between both hosts. Make sure SELinux/apparmor is disabled. ---
**Author:** ![Cris](https://avatars.discourse-cdn.com/v4/letter/c/71e660/32.png) [@Cris](https://forums.percona.com/u/Cris)\ **Post date:** [March 1, 2024, 6:25am UTC](https://forums.percona.com/t/unable-to-bootstrap-the-1st-node/28551/15 "2024-03-01T06:25:09Z")
Hi matthewb, Only port 4567 is in use by mysql as attached.SELINUX is DISABLED on both servers. ![Screenshot 2024-03-01 092310](https://us1.discourse-cdn.com/flex019/uploads/percona1/original/2X/3/34769c6fd999f487e8d776fb2427741d4fbe91da.png) ---
**Author:** ![matthewb](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/matthewb/32/34_2.png) [@matthewb](https://forums.percona.com/u/matthewb)\ **Post date:** [March 1, 2024, 4:32pm UTC](https://forums.percona.com/t/unable-to-bootstrap-the-1st-node/28551/16 "2024-03-01T16:32:35Z")
What about firewall? Is that disabled? (iptables/ufw, etc) ---
**Author:** ![Cris](https://avatars.discourse-cdn.com/v4/letter/c/71e660/32.png) [@Cris](https://forums.percona.com/u/Cris)\ **Post date:** [March 5, 2024, 1:09pm UTC](https://forums.percona.com/t/unable-to-bootstrap-the-1st-node/28551/17 "2024-03-05T13:09:20Z")
They are all disabled on all the nodes. ---
**Author:** ![matthewb](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/matthewb/32/34_2.png) [@matthewb](https://forums.percona.com/u/matthewb)\ **Post date:** [March 5, 2024, 4:01pm UTC](https://forums.percona.com/t/unable-to-bootstrap-the-1st-node/28551/18 "2024-03-05T16:01:13Z")
Is node1 in PRIMARY state? Able to read/write data? ---
**Author:** ![Cris](https://avatars.discourse-cdn.com/v4/letter/c/71e660/32.png) [@Cris](https://forums.percona.com/u/Cris)\ **Post date:** [March 6, 2024, 5:31am UTC](https://forums.percona.com/t/unable-to-bootstrap-the-1st-node/28551/19 "2024-03-06T05:31:09Z")
Hi Matthewb, Node 1 is in primary and can read/write data as shown below ![Screenshot 2024-03-06 082923](https://us1.discourse-cdn.com/flex019/uploads/percona1/original/2X/2/24b302f19f327c4618e6b62a97f186fc56c2a3ab.png) ---
**Author:** ![matthewb](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/matthewb/32/34_2.png) [@matthewb](https://forums.percona.com/u/matthewb)\ **Post date:** [March 6, 2024, 4:21pm UTC](https://forums.percona.com/t/unable-to-bootstrap-the-1st-node/28551/20 "2024-03-06T16:21:22Z")
So node1 is online, bootstrapped, and in PRIMARY state. But when you try to start node2, it will not join. 9/10 times this is network related. Read over your my.cnf again and check everything lines up. Provide both here if you wish.