Not the answer you need?
Register and ask your own question!

PXC-8.0 --> Unable to start the 2nd node after bootstrap

AneeshBabuAneeshBabu Current User Role Supporter

Hi there ..

Unable to start the second node after boot strapping the 1st node.

attaching the error log


[email protected]:~# service mysql restart

Job for mysql.service failed because the control process exited with error code.

See "systemctl status mysql.service" and "journalctl -xe" for details.

[email protected]:~# systemctl status mysql.service

● mysql.service - Percona XtraDB Cluster

  Loaded: loaded (/lib/systemd/system/mysql.service; enabled; vendor preset: enabled)

  Active: failed (Result: exit-code) since Wed 2020-10-28 14:07:19 IST; 9s ago

 Process: 30073 ExecStopPost=/usr/bin/mysql-systemd stop-post (code=exited, status=0/SUCCESS)

 Process: 3748 ExecStop=/usr/bin/mysql-systemd stop (code=exited, status=0/SUCCESS)

 Process: 2375 ExecStartPost=/usr/bin/mysql-systemd start-post $MAINPID (code=exited, status=0/SUCCESS)

 Process: 2373 ExecStartPost=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)

 Process: 30068 ExecStart=/usr/sbin/mysqld $_WSREP_START_POSITION (code=exited, status=1/FAILURE)

 Process: 30019 ExecStartPre=/bin/sh -c VAR=`bash /usr/bin/mysql-systemd galera-recovery`; [ $? -eq 0 ] && systemctl set-environment _WSREP_START_POSITION

 Process: 30014 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)

 Process: 29983 ExecStartPre=/usr/bin/mysql-systemd check-grastate (code=exited, status=0/SUCCESS)

 Process: 29934 ExecStartPre=/usr/bin/mysql-systemd start-pre (code=exited, status=0/SUCCESS)

 Main PID: 30068 (code=exited, status=1/FAILURE)

  Status: "Server startup in progress"


Oct 28 14:06:45 pxc-2 systemd[1]: Starting Percona XtraDB Cluster...

Oct 28 14:07:19 pxc-2 systemd[1]: mysql.service: Main process exited, code=exited, status=1/FAILURE

Oct 28 14:07:19 pxc-2 mysql-systemd[30073]: WARNING: mysql pid file /var/run/mysqld/mysqld.pid empty or not readable

Oct 28 14:07:19 pxc-2 mysql-systemd[30073]: WARNING: mysql may be already dead

Oct 28 14:07:19 pxc-2 systemd[1]: mysql.service: Failed with result 'exit-code'.

Oct 28 14:07:19 pxc-2 systemd[1]: Failed to start Percona XtraDB Cluster.

lines 1-21/21 (END)

[3]+ Stopped                systemctl status mysql.service


Tagged:

Answers

  • matthewbmatthewb Senior [email protected] Percona Staff Role

    Hi @AneeshBabu Please read your log file. It is full of connection timeout messages. Ensure there is no firewall/iptables blocking ports 4444 and 4567 between your nodes. PXC8 bootstraps with SSL enabled by default. Did you copy the SSL certificates from node1 over to node2 before starting it?

  • Evgeniy PatlanEvgeniy Patlan Percona Percona Staff Role

    Hi @AneeshBabu

    I tried to reproduce your issue but without success.

    Please try one more time following instructions here: https://www.percona.com/doc/percona-xtradb-cluster/LATEST/configure.html#configure

  • AneeshBabuAneeshBabu Current User Role Supporter

    Hi there ....

    @matthewb @Evgeniy Patlan

    When bootstrapped the node 1 facing the below error.Attaching the error log file of node 1.


    [email protected]:~# systemctl start [email protected]
    Job for [email protected] failed because the control process exited with error code.
    See "systemctl status [email protected]" and "journalctl -xe" for details.
    
    
    [email protected]:~# systemctl status [email protected][email protected] - Percona XtraDB Cluster with config /etc/default/mysql.bootstrap
       Loaded: loaded (/lib/systemd/system/[email protected]; disabled; vendor preset: enabled)
       Active: failed (Result: exit-code) since Mon 2020-11-02 23:22:01 IST; 10min ago
      Process: 8706 ExecStopPost=/usr/bin/mysql-systemd stop-post (code=exited, status=0/SUCCESS)
      Process: 8704 ExecStart=/usr/sbin/mysqld $EXTRA_ARGS $_WSREP_START_POSITION (code=exited, status=1/FAILURE)
      Process: 8656 ExecStartPre=/bin/sh -c VAR=`bash /usr/bin/mysql-systemd galera-recovery`; [ $? -eq 0 ] && systemctl set-environment _WSREP_START_POSITION=$VAR || exit 1 (code=exited, s
      Process: 8654 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
      Process: 8617 ExecStartPre=/usr/bin/mysql-systemd start-pre (code=exited, status=0/SUCCESS)
     Main PID: 8704 (code=exited, status=1/FAILURE)
       Status: "Server startup in progress"
    
    Nov 02 23:22:00 pxc-1 systemd[1]: Starting Percona XtraDB Cluster with config /etc/default/mysql.bootstrap...
    Nov 02 23:22:01 pxc-1 systemd[1]: [email protected]: Main process exited, code=exited, status=1/FAILURE
    Nov 02 23:22:01 pxc-1 mysql-systemd[8706]:  WARNING: mysql pid file /var/run/mysqld/mysqld.pid empty or not readable
    Nov 02 23:22:01 pxc-1 mysql-systemd[8706]:  WARNING: mysql may be already dead
    Nov 02 23:22:01 pxc-1 systemd[1]: [email protected]: Failed with result 'exit-code'.
    Nov 02 23:22:01 pxc-1 systemd[1]: Failed to start Percona XtraDB Cluster with config /etc/default/mysql.bootstrap.
    


  • matthewbmatthewb Senior [email protected] Percona Staff Role

    @AneeshBabu Please attempt some basic troubleshooting yourself. I read in your log file this line:

    2020-11-02T23:38:25.859782+05:30 0 [ERROR] [MY-000000] [Galera] It may not be safe to bootstrap the cluster from this node. It was not the last one to leave the cluster and may not contain all the updates. To force cluster bootstrap with this node, edit the grastate.dat file manually and set safe_to_bootstrap to 1 .

    This clearly tells you the problem and tells you what to do.

  • AneeshBabuAneeshBabu Current User Role Supporter
    edited November 3

    Hi @matthewb

    Boot strapped node 1 ,it is working fine.

    when started the other nodes i.e node 2 & 3 facing the below error.Could not be able to create the PID file.

    i.e

    Nov 03 23:04:00 pxc-2 mysql-systemd[2804]: WARNING: mysql pid file /var/run/mysqld/mysqld.pid empty or not readable

    Nov 03 23:04:00 pxc-2 mysql-systemd[2804]: WARNING: mysql may be already dead


    Note : both the nodes having ---- safe_to_bootstrap: 0 in grastate.dat file .


    Attaching the error log file.

    [email protected]:~# service mysql start
    Job for mysql.service failed because the control process exited with error code.
    See "systemctl status mysql.service" and "journalctl -xe" for details.
    
    [email protected]:~# systemctl status mysql.service
    ● mysql.service - Percona XtraDB Cluster
       Loaded: loaded (/lib/systemd/system/mysql.service; enabled; vendor preset: enabled)
       Active: failed (Result: exit-code) since Tue 2020-11-03 23:04:00 IST; 21s ago
      Process: 2804 ExecStopPost=/usr/bin/mysql-systemd stop-post (code=exited, status=0/SUCCESS)
      Process: 2799 ExecStart=/usr/sbin/mysqld $_WSREP_START_POSITION (code=exited, status=1/FAILURE)
      Process: 2751 ExecStartPre=/bin/sh -c VAR=`bash /usr/bin/mysql-systemd galera-recovery`; [ $? -eq 0 ] && systemctl set-environment _WSREP_START_POSITION=
      Process: 2745 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
      Process: 2714 ExecStartPre=/usr/bin/mysql-systemd check-grastate (code=exited, status=0/SUCCESS)
      Process: 2665 ExecStartPre=/usr/bin/mysql-systemd start-pre (code=exited, status=0/SUCCESS)
     Main PID: 2799 (code=exited, status=1/FAILURE)
       Status: "Server startup in progress"
        Error: 11 (Resource temporarily unavailable)
    
    Nov 03 23:04:00 pxc-2 systemd[1]: Starting Percona XtraDB Cluster...
    Nov 03 23:04:00 pxc-2 systemd[1]: mysql.service: Main process exited, code=exited, status=1/FAILURE
    Nov 03 23:04:00 pxc-2 mysql-systemd[2804]:  WARNING: mysql pid file /var/run/mysqld/mysqld.pid empty or not readable
    Nov 03 23:04:00 pxc-2 mysql-systemd[2804]:  WARNING: mysql may be already dead
    Nov 03 23:04:00 pxc-2 systemd[1]: mysql.service: Failed with result 'exit-code'.
    Nov 03 23:04:00 pxc-2 systemd[1]: Failed to start Percona XtraDB Cluster.
    lines 1-19/19 (END)
    [5]+  Stopped                 systemctl status mysql.service
    


  • matthewbmatthewb Senior [email protected] Percona Staff Role

    You have something messed up in your my.cnf. What do you have for 'wsrep_cluster_address'?

    [Galera] Failed to resolve tcp://gcomm:4567
    2020-11-03T23:04:00.632294+05:30 0 [Note] [MY-000000] [Galera] (c2733461, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567
    2020-11-03T23:04:00.632484+05:30 0 [Note] [MY-000000] [Galera] (c2733461, 'tcp://0.0.0.0:4567') multicast: , ttl: 1
    2020-11-03T23:04:00.632957+05:30 0 [Note] [MY-000000] [Galera] EVS version 1
    2020-11-03T23:04:00.633283+05:30 0 [Note] [MY-000000] [Galera] gcomm: connecting to group 'pxc-cluster', peer 'gcomm:'
    2020-11-03T23:04:00.633701+05:30 0 [ERROR] [MY-000000] [Galera] failed to open gcomm backend connection: 131: No address to connect (FATAL)
    
  • AneeshBabuAneeshBabu Current User Role Supporter

    Hi @matthewb

    node 2 also working fine .Could not start the node 3

    Node : 3 config file

    [client]
    socket=/var/run/mysqld/mysqld.sock
    
    [mysqld]
    server-id                       =       1
    datadir                         =       /var/lib/mysql
    socket                          =       /var/run/mysqld/mysqld.sock
    log-error                       =       /var/log/mysql/error.log
    pid-file                        =       /var/run/mysqld/mysqld.pid
    binlog_expire_logs_seconds      =       604800
    
    wsrep_provider                  =       /usr/lib/galera4/libgalera_smm.so
    binlog_format                   =       ROW
    wsrep_slave_threads             =       8
    wsrep_log_conflicts
    innodb_autoinc_lock_mode        =       2
    
    log_timestamps                  =       SYSTEM
    
    # Node IP address
    wsrep_node_address              =       192.168.0.1
    wsrep_cluster_address           =       gcomm://192.168.0.1,192.168.0.2,192.168.0.3
    wsrep_cluster_name              =       pxc-cluster
    wsrep_node_name                 =       pxc3
    pxc_strict_mode                 =       ENFORCING
    wsrep_sst_method                =       xtrabackup-v2
    pxc-encrypt-cluster-traffic     =       ON
    


  • matthewbmatthewb Senior [email protected] Percona Staff Role

    Node3's Ip address is 192.168.0.1?

    Also, each should have different server_id's too.

Sign In or Register to comment.

MySQL, InnoDB, MariaDB and MongoDB are trademarks of their respective owners.
Copyright ©2005 - 2020 Percona LLC. All rights reserved.