PXC-8.0 --> Unable to start the 2nd node after bootstrap

Hi there …

Unable to start the second node after boot strapping the 1st node.

attaching the error log

PXC-8.0-Node2 Error.log

root@pxc-2:~# service mysql restart

Job for mysql.service failed because the control process exited with error code.

See "systemctl status mysql.service" and "journalctl -xe" for details.

root@pxc-2:~# systemctl status mysql.service

● mysql.service - Percona XtraDB Cluster

  Loaded: loaded (/lib/systemd/system/mysql.service; enabled; vendor preset: enabled)

  Active: failed (Result: exit-code) since Wed 2020-10-28 14:07:19 IST; 9s ago

 Process: 30073 ExecStopPost=/usr/bin/mysql-systemd stop-post (code=exited, status=0/SUCCESS)

 Process: 3748 ExecStop=/usr/bin/mysql-systemd stop (code=exited, status=0/SUCCESS)

 Process: 2375 ExecStartPost=/usr/bin/mysql-systemd start-post $MAINPID (code=exited, status=0/SUCCESS)

 Process: 2373 ExecStartPost=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)

 Process: 30068 ExecStart=/usr/sbin/mysqld $_WSREP_START_POSITION (code=exited, status=1/FAILURE)

 Process: 30019 ExecStartPre=/bin/sh -c VAR=`bash /usr/bin/mysql-systemd galera-recovery`; [ $? -eq 0 ] && systemctl set-environment _WSREP_START_POSITION

 Process: 30014 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)

 Process: 29983 ExecStartPre=/usr/bin/mysql-systemd check-grastate (code=exited, status=0/SUCCESS)

 Process: 29934 ExecStartPre=/usr/bin/mysql-systemd start-pre (code=exited, status=0/SUCCESS)

 Main PID: 30068 (code=exited, status=1/FAILURE)

  Status: "Server startup in progress"


Oct 28 14:06:45 pxc-2 systemd[1]: Starting Percona XtraDB Cluster...

Oct 28 14:07:19 pxc-2 systemd[1]: mysql.service: Main process exited, code=exited, status=1/FAILURE

Oct 28 14:07:19 pxc-2 mysql-systemd[30073]: WARNING: mysql pid file /var/run/mysqld/mysqld.pid empty or not readable

Oct 28 14:07:19 pxc-2 mysql-systemd[30073]: WARNING: mysql may be already dead

Oct 28 14:07:19 pxc-2 systemd[1]: mysql.service: Failed with result 'exit-code'.

Oct 28 14:07:19 pxc-2 systemd[1]: Failed to start Percona XtraDB Cluster.

lines 1-21/21 (END)

[3]+ Stopped                systemctl status mysql.service

PXC-8.0-Node2 Error.log (17.6 KB)

Hi @AneeshBabu Please read your log file. It is full of connection timeout messages. Ensure there is no firewall/iptables blocking ports 4444 and 4567 between your nodes. PXC8 bootstraps with SSL enabled by default. Did you copy the SSL certificates from node1 over to node2 before starting it?

1 Like

Hi @AneeshBabu

I tried to reproduce your issue but without success.

Please try one more time following instructions here: https://www.percona.com/doc/percona-xtradb-cluster/LATEST/configure.html#configure

Hi there …

@matthewb @Evgeniy_Patlan

When bootstrapped the node 1 facing the below error.Attaching the error log file of node 1.

pxc-node1-error.log

root@pxc-1:~# systemctl start mysql@bootstrap.service
Job for mysql@bootstrap.service failed because the control process exited with error code.
See "systemctl status mysql@bootstrap.service" and "journalctl -xe" for details.


root@pxc-1:~# systemctl status mysql@bootstrap.service
● mysql@bootstrap.service - Percona XtraDB Cluster with config /etc/default/mysql.bootstrap
   Loaded: loaded (/lib/systemd/system/mysql@.service; disabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Mon 2020-11-02 23:22:01 IST; 10min ago
  Process: 8706 ExecStopPost=/usr/bin/mysql-systemd stop-post (code=exited, status=0/SUCCESS)
  Process: 8704 ExecStart=/usr/sbin/mysqld $EXTRA_ARGS $_WSREP_START_POSITION (code=exited, status=1/FAILURE)
  Process: 8656 ExecStartPre=/bin/sh -c VAR=`bash /usr/bin/mysql-systemd galera-recovery`; [ $? -eq 0 ] && systemctl set-environment _WSREP_START_POSITION=$VAR || exit 1 (code=exited, s
  Process: 8654 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
  Process: 8617 ExecStartPre=/usr/bin/mysql-systemd start-pre (code=exited, status=0/SUCCESS)
 Main PID: 8704 (code=exited, status=1/FAILURE)
   Status: "Server startup in progress"

Nov 02 23:22:00 pxc-1 systemd[1]: Starting Percona XtraDB Cluster with config /etc/default/mysql.bootstrap...
Nov 02 23:22:01 pxc-1 systemd[1]: mysql@bootstrap.service: Main process exited, code=exited, status=1/FAILURE
Nov 02 23:22:01 pxc-1 mysql-systemd[8706]:  WARNING: mysql pid file /var/run/mysqld/mysqld.pid empty or not readable
Nov 02 23:22:01 pxc-1 mysql-systemd[8706]:  WARNING: mysql may be already dead
Nov 02 23:22:01 pxc-1 systemd[1]: mysql@bootstrap.service: Failed with result 'exit-code'.
Nov 02 23:22:01 pxc-1 systemd[1]: Failed to start Percona XtraDB Cluster with config /etc/default/mysql.bootstrap.

@AneeshBabu Please attempt some basic troubleshooting yourself. I read in your log file this line:

2020-11-02T23:38:25.859782+05:30 0 [ERROR] [MY-000000] [Galera] It may not be safe to bootstrap the cluster from this node. It was not the last one to leave the cluster and may not contain all the updates. To force cluster bootstrap with this node, edit the grastate.dat file manually and set safe_to_bootstrap to 1 .

This clearly tells you the problem and tells you what to do.

Hi @matthewb

Boot strapped node 1 ,it is working fine.

when started the other nodes i.e node 2 & 3 facing the below error.Could not be able to create the PID file.

i.e

Nov 03 23:04:00 pxc-2 mysql-systemd[2804]: WARNING: mysql pid file /var/run/mysqld/mysqld.pid empty or not readable

Nov 03 23:04:00 pxc-2 mysql-systemd[2804]: WARNING: mysql may be already dead

Note : both the nodes having ---- safe_to_bootstrap: 0 in grastate.dat file .

Attaching the error log file.

node 2 error.log

root@pxc-2:~# service mysql start
Job for mysql.service failed because the control process exited with error code.
See "systemctl status mysql.service" and "journalctl -xe" for details.

root@pxc-2:~# systemctl status mysql.service
● mysql.service - Percona XtraDB Cluster
   Loaded: loaded (/lib/systemd/system/mysql.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Tue 2020-11-03 23:04:00 IST; 21s ago
  Process: 2804 ExecStopPost=/usr/bin/mysql-systemd stop-post (code=exited, status=0/SUCCESS)
  Process: 2799 ExecStart=/usr/sbin/mysqld $_WSREP_START_POSITION (code=exited, status=1/FAILURE)
  Process: 2751 ExecStartPre=/bin/sh -c VAR=`bash /usr/bin/mysql-systemd galera-recovery`; [ $? -eq 0 ] && systemctl set-environment _WSREP_START_POSITION=
  Process: 2745 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
  Process: 2714 ExecStartPre=/usr/bin/mysql-systemd check-grastate (code=exited, status=0/SUCCESS)
  Process: 2665 ExecStartPre=/usr/bin/mysql-systemd start-pre (code=exited, status=0/SUCCESS)
 Main PID: 2799 (code=exited, status=1/FAILURE)
   Status: "Server startup in progress"
    Error: 11 (Resource temporarily unavailable)

Nov 03 23:04:00 pxc-2 systemd[1]: Starting Percona XtraDB Cluster...
Nov 03 23:04:00 pxc-2 systemd[1]: mysql.service: Main process exited, code=exited, status=1/FAILURE
Nov 03 23:04:00 pxc-2 mysql-systemd[2804]:  WARNING: mysql pid file /var/run/mysqld/mysqld.pid empty or not readable
Nov 03 23:04:00 pxc-2 mysql-systemd[2804]:  WARNING: mysql may be already dead
Nov 03 23:04:00 pxc-2 systemd[1]: mysql.service: Failed with result 'exit-code'.
Nov 03 23:04:00 pxc-2 systemd[1]: Failed to start Percona XtraDB Cluster.
lines 1-19/19 (END)
[5]+  Stopped                 systemctl status mysql.service

You have something messed up in your my.cnf. What do you have for ‘wsrep_cluster_address’?

[Galera] Failed to resolve tcp://gcomm:4567

2020-11-03T23:04:00.632294+05:30 0 [Note] [MY-000000] [Galera] (c2733461, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567

2020-11-03T23:04:00.632484+05:30 0 [Note] [MY-000000] [Galera] (c2733461, 'tcp://0.0.0.0:4567') multicast: , ttl: 1

2020-11-03T23:04:00.632957+05:30 0 [Note] [MY-000000] [Galera] EVS version 1

2020-11-03T23:04:00.633283+05:30 0 [Note] [MY-000000] [Galera] gcomm: connecting to group 'pxc-cluster', peer 'gcomm:'

2020-11-03T23:04:00.633701+05:30 0 [ERROR] [MY-000000] [Galera] failed to open gcomm backend connection: 131: No address to connect (FATAL)

Hi @matthewbnode 2 also working fine .Could not start the node 3

Node : 3 config file

[client]
socket=/var/run/mysqld/mysqld.sock

[mysqld]
server-id                       =       1
datadir                         =       /var/lib/mysql
socket                          =       /var/run/mysqld/mysqld.sock
log-error                       =       /var/log/mysql/error.log
pid-file                        =       /var/run/mysqld/mysqld.pid
binlog_expire_logs_seconds      =       604800

wsrep_provider                  =       /usr/lib/galera4/libgalera_smm.so
binlog_format                   =       ROW
wsrep_slave_threads             =       8
wsrep_log_conflicts
innodb_autoinc_lock_mode        =       2

log_timestamps                  =       SYSTEM

# Node IP address
wsrep_node_address              =       192.168.0.1
wsrep_cluster_address           =       gcomm://192.168.0.1,192.168.0.2,192.168.0.3
wsrep_cluster_name              =       pxc-cluster
wsrep_node_name                 =       pxc3
pxc_strict_mode                 =       ENFORCING
wsrep_sst_method                =       xtrabackup-v2
pxc-encrypt-cluster-traffic     =       ON

Node3’s Ip address is 192.168.0.1?

Also, each should have different server_id’s too.