Hello,
Doesn’t matter whatever I do I can’t start Percona Node 2 in my 2 node cluster… (yes, I know that I should not run 2 nodes in a cluster but that’s what it is for now)
RHEL7, latest Percona 5.7.
Your time and help are much appropriated. This is a UAT setup but if all goes well it will go into PROD…
This setup was running fine until one day it did that
2017-11-17T11:55:13.044889Z 4 [Note] WSREP: Current view of cluster as seen by this node
view (view_id(NON_PRIM,5fc8c4bc,4)
memb {
8567ecca,0
}
joined {
}
left {
}
partitioned {
5fc8c4bc,0
}
)
2017-11-17T11:55:13.044957Z 4 [Note] WSREP: Current view of cluster as seen by this node
view ((empty))
2017-11-17T11:55:13.045119Z 4 [Note] WSREP: gcomm: closed
2017-11-17T11:55:13.045186Z 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
2017-11-17T11:55:13.045220Z 0 [Note] WSREP: Flow-control interval: [100, 100]
2017-11-17T11:55:13.045225Z 0 [Note] WSREP: Trying to continue unpaused monitor
2017-11-17T11:55:13.045227Z 0 [Note] WSREP: Received NON-PRIMARY.
2017-11-17T11:55:13.045230Z 0 [Note] WSREP: Shifting SYNCED -> OPEN (TO: 8725)
2017-11-17T11:55:13.045250Z 0 [Note] WSREP: Received self-leave message.
2017-11-17T11:55:13.045254Z 0 [Note] WSREP: Flow-control interval: [0, 0]
2017-11-17T11:55:13.045257Z 0 [Note] WSREP: Trying to continue unpaused monitor
2017-11-17T11:55:13.045259Z 0 [Note] WSREP: Received SELF-LEAVE. Closing connection.
2017-11-17T11:55:13.045261Z 0 [Note] WSREP: Shifting OPEN -> CLOSED (TO: 8725)
2017-11-17T11:55:13.045284Z 0 [Note] WSREP: RECV thread exiting 0: Success
2017-11-17T11:55:13.045345Z 4 [Note] WSREP: recv_thread() joined.
2017-11-17T11:55:13.045353Z 4 [Note] WSREP: Closing replication queue.
2017-11-17T11:55:13.045357Z 4 [Note] WSREP: Closing slave action queue.
2017-11-17T11:55:13.045756Z 4 [Note] WSREP: /usr/sbin/mysqld: Terminated.
Nothing gets logged to the /var/log/mysql.log
Here is my error message:
-- Unit mysql.service has begun starting up.
Nov 24 13:38:27 S-02 mysqld_safe[126039]: 2017-11-24T13:38:27.619589Z mysqld_safe Logging to '/var/log/mysqld-safe.log'.
Nov 24 13:38:27 S-02 mysqld_safe[126039]: 2017-11-24T13:38:27.623103Z mysqld_safe Logging to '/var/log/mysqld-safe.log'.
Nov 24 13:38:27 S-02 mysqld_safe[126039]: 2017-11-24T13:38:27.649402Z mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
Nov 24 13:38:27 S-02 mysqld_safe[126039]: 2017-11-24T13:38:27.665308Z mysqld_safe WSREP: Running position recovery with --log_error='/var/lib/mysql/wsrep_recovery.pZnH0L' --pid-file='/var/lib/mysql/server-02-recover.pid'
Nov 24 13:38:31 S-02 mysql-systemd[126040]: /usr/bin/mysql-systemd: line 140: kill: (126039) - No such process
Nov 24 13:38:31 S-02 mysql-systemd[126040]: ERROR! mysqld_safe with PID 126039 has already exited: FAILURE
Nov 24 13:38:31 S-02 systemd[1]: mysql.service: control process exited, code=exited status=1
Nov 24 13:38:31 S-02 mysql-systemd[126701]: WARNING: mysql pid file /var/lib/mysql/mysqld.pid empty or not readable
Nov 24 13:38:31 S-02 mysql-systemd[126701]: ERROR! mysql already dead
Nov 24 13:38:31 S-02 systemd[1]: mysql.service: control process exited, code=exited status=2
Nov 24 13:38:31 S-02 mysql-systemd[126732]: WARNING: mysql pid file /var/lib/mysql/mysqld.pid empty or not readable
Nov 24 13:38:31 S-02 mysql-systemd[126732]: WARNING: mysql may be already dead
Nov 24 13:38:31 S-02 systemd[1]: Failed to start Percona XtraDB Cluster.
-- Subject: Unit mysql.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit mysql.service has failed.
--
-- The result is failed.
Nov 24 13:38:31 S-02 systemd[1]: Unit mysql.service entered failed state.
Nov 24 13:38:31 S-02 systemd[1]: mysql.service failed.
Nov 24 13:38:31 S-02 polkitd[622]: Unregistered Authentication Agent for unix-process:125992:169691973 (system bus name :1.22171, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8) (disconnected from bus)
And that’s what in wsrep_recovery:
2017-11-24T13:38:27.847539Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2017-11-24T13:38:27.849040Z 0 [Note] /usr/sbin/mysqld (mysqld 5.7.19-17-57-log) starting as process 126662 ...
2017-11-24T13:38:27.852025Z 0 [Warning] No argument was provided to --log-bin, and --log-bin-index was not used; so replication may break when this MySQL server acts as a master and has his hostname changed!! Please use '--log-bin=S-02-bin' to avoid this problem.
2017-11-24T13:38:27.856882Z 0 [Note] InnoDB: PUNCH HOLE support available
[...CUT...]
2017-11-24T13:38:28.523347Z 0 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.7.19-17 started; log sequence number 126001947
2017-11-24T13:38:28.523365Z 0 [Warning] InnoDB: Skipping buffer pool dump/restore during wsrep recovery.
2017-11-24T13:38:28.523658Z 0 [Note] Plugin 'FEDERATED' is disabled.
2017-11-24T13:38:28.549281Z 0 [Note] WSREP: Recovered position: 601f3fef-c30e-11e7-81ab-f27164665fa9:8724
2017-11-24T13:38:28.549313Z 0 [Note] Binlog end
2017-11-24T13:38:28.554438Z 0 [Note] Shutting down plugin 'validate_password'
2017-11-24T13:38:28.554458Z 0 [Note] Shutting down plugin 'ngram'
2017-11-24T13:38:28.554460Z 0 [Note] Shutting down plugin 'partition'
2017-11-24T13:38:28.554462Z 0 [Note] Shutting down plugin 'BLACKHOLE'
2017-11-24T13:38:28.554464Z 0 [Note] Shutting down plugin 'ARCHIVE'
2017-11-24T13:38:28.554466Z 0 [Note] Shutting down plugin 'INNODB_SYS_VIRTUAL'
2017-11-24T13:38:28.554468Z 0 [Note] Shutting down plugin 'INNODB_CHANGED_PAGES'
2017-11-24T13:38:28.554469Z 0 [Note] Shutting down plugin 'INNODB_SYS_DATAFILES'
2017-11-24T13:38:28.554471Z 0 [Note] Shutting down plugin 'INNODB_SYS_TABLESPACES'
[...CUT...]
2017-11-24T13:38:28.554520Z 0 [Note] Shutting down plugin 'XTRADB_ZIP_DICT_COLS'
2017-11-24T13:38:28.554522Z 0 [Note] Shutting down plugin 'XTRADB_ZIP_DICT'
2017-11-24T13:38:28.554524Z 0 [Note] Shutting down plugin 'XTRADB_RSEG'
2017-11-24T13:38:28.554525Z 0 [Note] Shutting down plugin 'XTRADB_INTERNAL_HASH_TABLES'
2017-11-24T13:38:28.554527Z 0 [Note] Shutting down plugin 'XTRADB_READ_VIEW'
2017-11-24T13:38:28.554528Z 0 [Note] Shutting down plugin 'InnoDB'
2017-11-24T13:38:28.554636Z 0 [Note] InnoDB: FTS optimize thread exiting.
2017-11-24T13:38:28.554738Z 0 [Note] InnoDB: Starting shutdown...
2017-11-24T13:38:29.656074Z 0 [Note] InnoDB: Waiting for page_cleaner to finish flushing of buffer pool
2017-11-24T13:38:30.568578Z 0 [Note] InnoDB: Shutdown completed; log sequence number 126001966
2017-11-24T13:38:30.570358Z 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
2017-11-24T13:38:30.570371Z 0 [Note] Shutting down plugin 'PERFORMANCE_SCHEMA'
2017-11-24T13:38:30.570420Z 0 [Note] Shutting down plugin 'MEMORY'
2017-11-24T13:38:30.570424Z 0 [Note] Shutting down plugin 'CSV'
2017-11-24T13:38:30.570428Z 0 [Note] Shutting down plugin 'MyISAM'
2017-11-24T13:38:30.570457Z 0 [Note] Shutting down plugin 'MRG_MYISAM'
2017-11-24T13:38:30.570459Z 0 [Note] Shutting down plugin 'sha256_password'
2017-11-24T13:38:30.570483Z 0 [Note] Shutting down plugin 'mysql_native_password'
2017-11-24T13:38:30.570485Z 0 [Note] Shutting down plugin 'wsrep'
2017-11-24T13:38:30.570658Z 0 [Note] Shutting down plugin 'binlog'
2017-11-24T13:38:30.570889Z 0 [Note] /usr/sbin/mysqld: Shutdown complete