Not the answer you need?
Register and ask your own question!

Can't start Percona Cluster (mysql) after "self-leave"

columbcolumb EntrantCurrent User Role Beginner
Hello,

Doesn't matter whatever I do I can't start Percona Node 2 in my 2 node cluster.... (yes, I know that I should not run 2 nodes in a cluster but that's what it is for now)
RHEL7, latest Percona 5.7.

Your time and help are much appropriated. This is a UAT setup but if all goes well it will go into PROD...

This setup was running fine until one day it did that
2017-11-17T11:55:13.044889Z 4 [Note] WSREP: Current view of cluster as seen by this node
view (view_id(NON_PRIM,5fc8c4bc,4)
memb {
        8567ecca,0
        }
joined {
        }
left {
        }
partitioned {
        5fc8c4bc,0
        }
)
2017-11-17T11:55:13.044957Z 4 [Note] WSREP: Current view of cluster as seen by this node
view ((empty))
2017-11-17T11:55:13.045119Z 4 [Note] WSREP: gcomm: closed
2017-11-17T11:55:13.045186Z 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
2017-11-17T11:55:13.045220Z 0 [Note] WSREP: Flow-control interval: [100, 100]
2017-11-17T11:55:13.045225Z 0 [Note] WSREP: Trying to continue unpaused monitor
2017-11-17T11:55:13.045227Z 0 [Note] WSREP: Received NON-PRIMARY.
2017-11-17T11:55:13.045230Z 0 [Note] WSREP: Shifting SYNCED -> OPEN (TO: 8725)
2017-11-17T11:55:13.045250Z 0 [Note] WSREP: Received self-leave message.
2017-11-17T11:55:13.045254Z 0 [Note] WSREP: Flow-control interval: [0, 0]
2017-11-17T11:55:13.045257Z 0 [Note] WSREP: Trying to continue unpaused monitor
2017-11-17T11:55:13.045259Z 0 [Note] WSREP: Received SELF-LEAVE. Closing connection.
2017-11-17T11:55:13.045261Z 0 [Note] WSREP: Shifting OPEN -> CLOSED (TO: 8725)
2017-11-17T11:55:13.045284Z 0 [Note] WSREP: RECV thread exiting 0: Success
2017-11-17T11:55:13.045345Z 4 [Note] WSREP: recv_thread() joined.
2017-11-17T11:55:13.045353Z 4 [Note] WSREP: Closing replication queue.
2017-11-17T11:55:13.045357Z 4 [Note] WSREP: Closing slave action queue.
2017-11-17T11:55:13.045756Z 4 [Note] WSREP: /usr/sbin/mysqld: Terminated.

Nothing gets logged to the /var/log/mysql.log

Here is my error message:
-- Unit mysql.service has begun starting up.
Nov 24 13:38:27 S-02 mysqld_safe[126039]: 2017-11-24T13:38:27.619589Z mysqld_safe Logging to '/var/log/mysqld-safe.log'.
Nov 24 13:38:27 S-02 mysqld_safe[126039]: 2017-11-24T13:38:27.623103Z mysqld_safe Logging to '/var/log/mysqld-safe.log'.
Nov 24 13:38:27 S-02 mysqld_safe[126039]: 2017-11-24T13:38:27.649402Z mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
Nov 24 13:38:27 S-02 mysqld_safe[126039]: 2017-11-24T13:38:27.665308Z mysqld_safe WSREP: Running position recovery with --log_error='/var/lib/mysql/wsrep_recovery.pZnH0L' --pid-file='/var/lib/mysql/server-02-recover.pid'
Nov 24 13:38:31 S-02 mysql-systemd[126040]: /usr/bin/mysql-systemd: line 140: kill: (126039) - No such process
Nov 24 13:38:31 S-02 mysql-systemd[126040]: ERROR! mysqld_safe with PID 126039 has already exited: FAILURE
Nov 24 13:38:31 S-02 systemd[1]: mysql.service: control process exited, code=exited status=1
Nov 24 13:38:31 S-02 mysql-systemd[126701]: WARNING: mysql pid file /var/lib/mysql/mysqld.pid empty or not readable
Nov 24 13:38:31 S-02 mysql-systemd[126701]: ERROR! mysql already dead
Nov 24 13:38:31 S-02 systemd[1]: mysql.service: control process exited, code=exited status=2
Nov 24 13:38:31 S-02 mysql-systemd[126732]: WARNING: mysql pid file /var/lib/mysql/mysqld.pid empty or not readable
Nov 24 13:38:31 S-02 mysql-systemd[126732]: WARNING: mysql may be already dead
Nov 24 13:38:31 S-02 systemd[1]: Failed to start Percona XtraDB Cluster.
-- Subject: Unit mysql.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit mysql.service has failed.
--
-- The result is failed.
Nov 24 13:38:31 S-02 systemd[1]: Unit mysql.service entered failed state.
Nov 24 13:38:31 S-02 systemd[1]: mysql.service failed.
Nov 24 13:38:31 S-02 polkitd[622]: Unregistered Authentication Agent for unix-process:125992:169691973 (system bus name :1.22171, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8) (disconnected from bus)

And that's what in wsrep_recovery:
2017-11-24T13:38:27.847539Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2017-11-24T13:38:27.849040Z 0 [Note] /usr/sbin/mysqld (mysqld 5.7.19-17-57-log) starting as process 126662 ...
2017-11-24T13:38:27.852025Z 0 [Warning] No argument was provided to --log-bin, and --log-bin-index was not used; so replication may break when this MySQL server acts as a master and has his hostname changed!! Please use '--log-bin=S-02-bin' to avoid this problem.
2017-11-24T13:38:27.856882Z 0 [Note] InnoDB: PUNCH HOLE support available
[...CUT...]
2017-11-24T13:38:28.523347Z 0 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.7.19-17 started; log sequence number 126001947
2017-11-24T13:38:28.523365Z 0 [Warning] InnoDB: Skipping buffer pool dump/restore during wsrep recovery.
2017-11-24T13:38:28.523658Z 0 [Note] Plugin 'FEDERATED' is disabled.
2017-11-24T13:38:28.549281Z 0 [Note] WSREP: Recovered position: 601f3fef-c30e-11e7-81ab-f27164665fa9:8724
2017-11-24T13:38:28.549313Z 0 [Note] Binlog end
2017-11-24T13:38:28.554438Z 0 [Note] Shutting down plugin 'validate_password'
2017-11-24T13:38:28.554458Z 0 [Note] Shutting down plugin 'ngram'
2017-11-24T13:38:28.554460Z 0 [Note] Shutting down plugin 'partition'
2017-11-24T13:38:28.554462Z 0 [Note] Shutting down plugin 'BLACKHOLE'
2017-11-24T13:38:28.554464Z 0 [Note] Shutting down plugin 'ARCHIVE'
2017-11-24T13:38:28.554466Z 0 [Note] Shutting down plugin 'INNODB_SYS_VIRTUAL'
2017-11-24T13:38:28.554468Z 0 [Note] Shutting down plugin 'INNODB_CHANGED_PAGES'
2017-11-24T13:38:28.554469Z 0 [Note] Shutting down plugin 'INNODB_SYS_DATAFILES'
2017-11-24T13:38:28.554471Z 0 [Note] Shutting down plugin 'INNODB_SYS_TABLESPACES'
[...CUT...]
2017-11-24T13:38:28.554520Z 0 [Note] Shutting down plugin 'XTRADB_ZIP_DICT_COLS'
2017-11-24T13:38:28.554522Z 0 [Note] Shutting down plugin 'XTRADB_ZIP_DICT'
2017-11-24T13:38:28.554524Z 0 [Note] Shutting down plugin 'XTRADB_RSEG'
2017-11-24T13:38:28.554525Z 0 [Note] Shutting down plugin 'XTRADB_INTERNAL_HASH_TABLES'
2017-11-24T13:38:28.554527Z 0 [Note] Shutting down plugin 'XTRADB_READ_VIEW'
2017-11-24T13:38:28.554528Z 0 [Note] Shutting down plugin 'InnoDB'
2017-11-24T13:38:28.554636Z 0 [Note] InnoDB: FTS optimize thread exiting.
2017-11-24T13:38:28.554738Z 0 [Note] InnoDB: Starting shutdown...
2017-11-24T13:38:29.656074Z 0 [Note] InnoDB: Waiting for page_cleaner to finish flushing of buffer pool
2017-11-24T13:38:30.568578Z 0 [Note] InnoDB: Shutdown completed; log sequence number 126001966
2017-11-24T13:38:30.570358Z 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
2017-11-24T13:38:30.570371Z 0 [Note] Shutting down plugin 'PERFORMANCE_SCHEMA'
2017-11-24T13:38:30.570420Z 0 [Note] Shutting down plugin 'MEMORY'
2017-11-24T13:38:30.570424Z 0 [Note] Shutting down plugin 'CSV'
2017-11-24T13:38:30.570428Z 0 [Note] Shutting down plugin 'MyISAM'
2017-11-24T13:38:30.570457Z 0 [Note] Shutting down plugin 'MRG_MYISAM'
2017-11-24T13:38:30.570459Z 0 [Note] Shutting down plugin 'sha256_password'
2017-11-24T13:38:30.570483Z 0 [Note] Shutting down plugin 'mysql_native_password'
2017-11-24T13:38:30.570485Z 0 [Note] Shutting down plugin 'wsrep'
2017-11-24T13:38:30.570658Z 0 [Note] Shutting down plugin 'binlog'
2017-11-24T13:38:30.570889Z 0 [Note] /usr/sbin/mysqld: Shutdown complete

Comments

  • columbcolumb Entrant Current User Role Beginner
    Hello,

    I'm getting nowhere with this - any ideas would be more than appropriated.

    I think that I should at leas get something in the log files? Or maybe in the journal etc?

    Thnaks
    Chris
Sign In or Register to comment.

MySQL, InnoDB, MariaDB and MongoDB are trademarks of their respective owners.
Copyright ©2005 - 2020 Percona LLC. All rights reserved.