Percona XTRADB Cluester is down. It running for last 2 months without any issue.
After upgrade, stopped and does not start.
Disable systemctl and tried mannually
/usr/sbin/mysqld --user=mysql --wsrep_recover --log-error=/tmp/start.tmp.LGZkh4w6HH.log
Log:
2017-09-29T13:20:08.867241Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp se$
2017-09-29T13:20:08.868441Z 0 [Warning] Insecure configuration for --secure-file-priv: Location is accessible to all OS users. Consider choo$
2017-09-29T13:20:08.868467Z 0 [Note] /usr/sbin/mysqld (mysqld 5.7.19-17-57-log) starting as process 1455 …
2017-09-29T13:20:08.870109Z 0 [Warning] No argument was provided to --log-bin, and --log-bin-index was not used; so replication may break wh$
2017-09-29T13:20:08.871155Z 0 [Note] InnoDB: PUNCH HOLE support available
2017-09-29T13:20:08.871167Z 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2017-09-29T13:20:08.871171Z 0 [Note] InnoDB: Uses event mutexes
2017-09-29T13:20:08.871173Z 0 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2017-09-29T13:20:08.871176Z 0 [Note] InnoDB: Compressed tables use zlib 1.2.8
2017-09-29T13:20:08.871178Z 0 [Note] InnoDB: Using Linux native AIO
2017-09-29T13:20:08.871358Z 0 [Note] InnoDB: Number of pools: 1
2017-09-29T13:20:08.871445Z 0 [Note] InnoDB: Using CPU crc32 instructions
2017-09-29T13:20:08.872478Z 0 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M
2017-09-29T13:20:08.875897Z 0 [Note] InnoDB: Completed initialization of buffer pool
2017-09-29T13:20:08.877668Z 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See th$
2017-09-29T13:20:08.888771Z 0 [Note] InnoDB: Crash recovery did not find the parallel doublewrite buffer at /var/lib/mysql/xb_doublewrite
2017-09-29T13:20:08.889147Z 0 [Note] InnoDB: Highest supported file format is Barracuda.
2017-09-29T13:20:09.193426Z 0 [Note] InnoDB: Created parallel doublewrite buffer at /var/lib/mysql/xb_doublewrite, size 3932160 bytes
2017-09-29T13:20:09.202630Z 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2017-09-29T13:20:09.202687Z 0 [Note] InnoDB: Setting file ‘./ibtmp1’ size to 12 MB. Physically writing the file full; Please wait …
2017-09-29T13:20:09.336173Z 0 [Note] InnoDB: File ‘./ibtmp1’ size is now 12 MB.
2017-09-29T13:20:09.336916Z 0 [Note] InnoDB: 96 redo rollback segment(s) found. 96 redo rollback segment(s) are active.
2017-09-29T13:20:09.336926Z 0 [Note] InnoDB: 32 non-redo rollback segment(s) are active.
2017-09-29T13:20:09.337460Z 0 [Note] InnoDB: Waiting for purge to start
2017-09-29T13:20:09.387631Z 0 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.7.19-17 started; log sequence number 157903695
2017-09-29T13:20:09.387668Z 0 [Warning] InnoDB: Skipping buffer pool dump/restore during wsrep recovery.
2017-09-29T13:20:09.387891Z 0 [Note] Plugin ‘FEDERATED’ is disabled.
2017-09-29T13:20:09.581026Z 0 [Note] WSREP: Recovered position: dcb3260d-5d9a-11e7-b715-cf577114ed32:13578
2017-09-29T13:20:09.581047Z 0 [Note] Binlog end
2017-09-29T13:20:09.626804Z 0 [Note] Shutting down plugin ‘ngram’
2017-09-29T13:20:09.626831Z 0 [Note] Shutting down plugin ‘partition’
2017-09-29T13:20:09.626835Z 0 [Note] Shutting down plugin ‘BLACKHOLE’
2017-09-29T13:20:09.626838Z 0 [Note] Shutting down plugin ‘ARCHIVE’
2017-09-29T13:20:09.626840Z 0 [Note] Shutting down plugin ‘INNODB_SYS_VIRTUAL’
2017-09-29T13:20:09.626842Z 0 [Note] Shutting down plugin ‘INNODB_CHANGED_PAGES’
2017-09-29T13:20:09.626844Z 0 [Note] Shutting down plugin ‘INNODB_SYS_DATAFILES’
2017-09-29T13:20:09.626846Z 0 [Note] Shutting down plugin ‘INNODB_SYS_TABLESPACES’
2017-09-29T13:20:09.626848Z 0 [Note] Shutting down plugin ‘INNODB_SYS_FOREIGN_COLS’
2017-09-29T13:20:09.626851Z 0 [Note] Shutting down plugin ‘INNODB_SYS_FOREIGN’
2017-09-29T13:20:09.626853Z 0 [Note] Shutting down plugin ‘INNODB_SYS_FIELDS’
2017-09-29T13:20:09.626854Z 0 [Note] Shutting down plugin ‘INNODB_SYS_COLUMNS’
2017-09-29T13:20:09.626856Z 0 [Note] Shutting down plugin ‘INNODB_SYS_INDEXES’
2017-09-29T13:20:09.626868Z 0 [Note] Shutting down plugin ‘INNODB_SYS_TABLESTATS’
2017-09-29T13:20:09.626870Z 0 [Note] Shutting down plugin ‘INNODB_SYS_TABLES’
2017-09-29T13:20:09.626872Z 0 [Note] Shutting down plugin ‘INNODB_FT_INDEX_TABLE’
2017-09-29T13:20:09.626874Z 0 [Note] Shutting down plugin ‘INNODB_FT_INDEX_CACHE’
2017-09-29T13:20:09.626876Z 0 [Note] Shutting down plugin ‘INNODB_FT_CONFIG’
2017-09-29T13:20:09.626878Z 0 [Note] Shutting down plugin ‘INNODB_FT_BEING_DELETED’
2017-09-29T13:20:09.626880Z 0 [Note] Shutting down plugin ‘INNODB_FT_DELETED’
2017-09-29T13:20:09.626882Z 0 [Note] Shutting down plugin ‘INNODB_FT_DEFAULT_STOPWORD’
2017-09-29T13:20:09.626884Z 0 [Note] Shutting down plugin ‘INNODB_METRICS’
2017-09-29T13:20:09.626886Z 0 [Note] Shutting down plugin ‘INNODB_TEMP_TABLE_INFO’
2017-09-29T13:20:09.626888Z 0 [Note] Shutting down plugin ‘INNODB_BUFFER_POOL_STATS’
2017-09-29T13:20:09.626890Z 0 [Note] Shutting down plugin ‘INNODB_BUFFER_PAGE_LRU’
2017-09-29T13:20:09.626892Z 0 [Note] Shutting down plugin ‘INNODB_BUFFER_PAGE’
2017-09-29T13:20:09.626894Z 0 [Note] Shutting down plugin ‘INNODB_CMP_PER_INDEX_RESET’
2017-09-29T13:20:09.626896Z 0 [Note] Shutting down plugin ‘INNODB_CMP_PER_INDEX’
2017-09-29T13:20:09.626898Z 0 [Note] Shutting down plugin ‘INNODB_CMPMEM_RESET’
2017-09-29T13:20:09.626900Z 0 [Note] Shutting down plugin ‘INNODB_CMPMEM’
2017-09-29T13:20:09.626902Z 0 [Note] Shutting down plugin ‘INNODB_CMP_RESET’
2017-09-29T13:20:09.626904Z 0 [Note] Shutting down plugin ‘INNODB_CMP’
2017-09-29T13:20:09.626906Z 0 [Note] Shutting down plugin ‘INNODB_LOCK_WAITS’
2017-09-29T13:20:09.626908Z 0 [Note] Shutting down plugin ‘INNODB_LOCKS’
2017-09-29T13:20:09.626910Z 0 [Note] Shutting down plugin ‘INNODB_TRX’
2017-09-29T13:20:09.626912Z 0 [Note] Shutting down plugin ‘XTRADB_ZIP_DICT_COLS’
2017-09-29T13:20:09.626914Z 0 [Note] Shutting down plugin ‘XTRADB_ZIP_DICT’
2017-09-29T13:20:09.626916Z 0 [Note] Shutting down plugin ‘XTRADB_RSEG’
2017-09-29T13:20:09.626918Z 0 [Note] Shutting down plugin ‘XTRADB_INTERNAL_HASH_TABLES’
2017-09-29T13:20:09.626921Z 0 [Note] Shutting down plugin ‘XTRADB_READ_VIEW’
2017-09-29T13:20:09.626923Z 0 [Note] Shutting down plugin ‘InnoDB’
2017-09-29T13:20:09.626978Z 0 [Note] InnoDB: FTS optimize thread exiting.
2017-09-29T13:20:09.627092Z 0 [Note] InnoDB: Starting shutdown…
2017-09-29T13:20:10.528446Z 0 [Note] InnoDB: Waiting for page_cleaner to finish flushing of buffer pool
2017-09-29T13:20:11.581864Z 0 [Note] InnoDB: Shutdown completed; log sequence number 157903714
2017-09-29T13:20:11.583248Z 0 [Note] InnoDB: Removed temporary tablespace data file: “ibtmp1”
2017-09-29T13:20:11.583258Z 0 [Note] Shutting down plugin ‘MRG_MYISAM’
2017-09-29T13:20:11.583262Z 0 [Note] Shutting down plugin ‘CSV’
2017-09-29T13:20:11.583265Z 0 [Note] Shutting down plugin ‘MyISAM’
2017-09-29T13:20:11.583279Z 0 [Note] Shutting down plugin ‘PERFORMANCE_SCHEMA’
2017-09-29T13:20:11.583312Z 0 [Note] Shutting down plugin ‘MEMORY’
2017-09-29T13:20:11.583315Z 0 [Note] Shutting down plugin ‘sha256_password’
2017-09-29T13:20:11.583317Z 0 [Note] Shutting down plugin ‘mysql_native_password’
2017-09-29T13:20:11.583319Z 0 [Note] Shutting down plugin ‘wsrep’
2017-09-29T13:20:11.583512Z 0 [Note] Shutting down plugin ‘binlog’
2017-09-29T13:20:11.583661Z 0 [Note] /usr/sbin/mysqld: Shutdown complete
Same configuration that was perfectly working.