[root@pld1fmk0114 ~]# service mysql start --wsrep-sst-auth=sstuser:sstpass Stale sst_in_progress file in datadir Starting MySQL (Percona XtraDB Cluster)State transfer in progress, setting sleep higher ../usr/bin/mysqld_safe: line 220: 121854 Aborted nohup /usr/sbin/mysqld --basedir=/usr --datadir=/mysql/data --plugin-dir=/usr/lib64/mysql/plugin --user=mysql --wsrep-provider=/usr/lib64/galera3/libgalera_smm.so --wsrep-sst-auth=sstuser\:sstpass --log-error=/mysql/log/mysqld.log --pid-file=/mysql/mysqld.pid --socket=/mysql/mysql.sock --wsrep_start_position=00000000-0000-0000-0000-000000000000:-1 < /dev/null > /dev/null 2>&1 ERROR! The server quit without updating PID file (/mysql/mysqld.pid). ERROR! MySQL (Percona XtraDB Cluster) server startup failed! [root@pld1fmk0114 ~]# netstat -tlpn Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:4444 0.0.0.0:* LISTEN 122352/socat tcp 0 0 127.0.0.1:199 0.0.0.0:* LISTEN 2799/snmpd tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN 1877/rpcbind tcp 0 0 0.0.0.0:58641 0.0.0.0:* LISTEN 2482/rpc.statd tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 2816/sshd tcp 0 0 127.0.0.1:631 0.0.0.0:* LISTEN 2611/cupsd tcp 0 0 :::5989 :::* LISTEN 3080/cimserver tcp 0 0 :::111 :::* LISTEN 1877/rpcbind tcp 0 0 :::22 :::* LISTEN 2816/sshd tcp 0 0 ::1:631 :::* LISTEN 2611/cupsd tcp 0 0 :::46712 :::* LISTEN 2482/rpc.statd [root@pld1fmk0114 ~]# ifconfig eth0 Link encap:Ethernet HWaddr 00:50:56:94:5A:D9 inet addr:172.19.101.108 Bcast:172.19.101.255 Mask:255.255.255.0 inet6 addr: fe80::250:56ff:fe94:5ad9/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:2520687 errors:0 dropped:0 overruns:0 frame:0 TX packets:1909691 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:316321478 (301.6 MiB) TX bytes:198716459 (189.5 MiB) lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:65536 Metric:1 RX packets:189 errors:0 dropped:0 overruns:0 frame:0 TX packets:189 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:11004 (10.7 KiB) TX bytes:11004 (10.7 KiB) [root@pld1fmk0114 ~]# telnet 172.19.101.108 3306 Trying 172.19.101.108... telnet: connect to address 172.19.101.108: Connection refused telnet: Unable to connect to remote host: Connection refused [root@pld1fmk0114 ~]# telnet 172.19.101.107 3306 Trying 172.19.101.107... Connected to pld1fmk0113.jud.hksarg (172.19.101.107). Escape character is '^]'. OHost 'pld1fmk0114.jud.hksarg' is not allowed to connect to this MySQL serverConnection closed by foreign host. [root@pld1fmk0114 ~]# telnet 172.19.101.107 4444 Trying 172.19.101.107... telnet: connect to address 172.19.101.107: Connection refused telnet: Unable to connect to remote host: Connection refused [root@pld1fmk0114 ~]# telnet 172.19.101.107 4567 Trying 172.19.101.107... Connected to pld1fmk0113.jud.hksarg (172.19.101.107). Escape character is '^]'. $scO????ča???/?o??-???!?r}Connection closed by foreign host. [root@pld1fmk0114 ~]# telnet 172.19.101.107 4568 Trying 172.19.101.107... telnet: connect to address 172.19.101.107: Connection refused telnet: Unable to connect to remote host: Connection refused [root@pld1fmk0114 ~]# [root@pld1fmk0114 ~]# hostname pld1fmk0114 [root@pld1fmk0114 ~]# cat /etc/hosts 127.0.0.1 localhost.localdomain localhost.localdomain localhost ::1 localhost.localdomain localhost.localdomain localhost 172.19.52.107 pwbak0101.jud.hksarg pwbak0101 172.19.52.108 pwbak0102.jud.hksarg pwbak0102 172.20.52.167 pwbak0401.jud.hksarg pwbak0401 172.20.52.168 pwbak0402.jud.hksarg pwbak0402 10.33.179.109 pwf1bak0103.jud.hksarg pwf1bak0103 172.19.101.107 pld1fmk0113.jud.hksarg pld1fmk0113 172.19.101.108 pld1fmk0114.jud.hksarg pld1fmk0114 [root@pld1fmk0114 ~]# cat /etc/my.cnf [mysqld] server-id=2 datadir=/mysql/data socket=/mysql/mysql.sock log-error=/mysql/log/mysqld.log pid-file=/mysql/mysqld.pid log-bin=pld1fmk0114-bin log-bin-index=pld1fmk0114-bin.index max_binlog_size=500M log_slave_updates expire_logs_days=7 # Disabling symbolic-links is recommended to prevent assorted security risks symbolic-links=0 [mysqld_safe] pid-file = /mysql/mysqld.pid socket = /mysql/mysql.sock nice = 0 [mysqld] # Path to Galera library wsrep_provider=/usr/lib64/galera3/libgalera_smm.so table_open_cache=2000 explicit_defaults_for_timestamp = 1 #max_connections = 2000; # Cluster connection URL contains IPs of nodes #If no IP is found, this implies that a new cluster needs to be created, #in order to do that you need to bootstrap this node wsrep_cluster_address=gcomm://172.19.101.107,172.19.101.108 # In order for Galera to work correctly binlog format should be ROW binlog_format=ROW # MyISAM storage engine has only experimental support default_storage_engine=InnoDB # Slave thread to use wsrep_slave_threads= 8 wsrep_log_conflicts # This changes how InnoDB autoincrement locks are managed and is a requirement for Galera innodb_autoinc_lock_mode=2 # Node IP address wsrep_node_address=172.19.101.108:4567 wsrep_sst_receive_address=172.19.101.107 # Cluster name wsrep_cluster_name=fmk113114 #If wsrep_node_name is not specified, then system hostname will be used wsrep_node_name=pld1fmk0114 #pxc_strict_mode allowed values: DISABLED,PERMISSIVE,ENFORCING,MASTER pxc_strict_mode=PERMISSIVE # SST method wsrep_sst_donor=172.19.101.107 wsrep_sst_method=xtrabackup-v2 #Authentication for SST method wsrep_sst_auth="sstuser:sstpass" ==================================================================================================== /mysql/log/mysqld.log 2019-01-03T07:38:32.114602Z 0 [Note] /usr/sbin/mysqld (mysqld 5.7.23-23-57-log) starting as process 121818 ... 2019-01-03T07:38:33.794167Z 0 [Note] InnoDB: PUNCH HOLE support available 2019-01-03T07:38:33.794200Z 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2019-01-03T07:38:33.794216Z 0 [Note] InnoDB: Uses event mutexes 2019-01-03T07:38:33.794224Z 0 [Note] InnoDB: GCC builtin __sync_synchronize() is used for memory barrier 2019-01-03T07:38:33.794230Z 0 [Note] InnoDB: Compressed tables use zlib 1.2.3 2019-01-03T07:38:33.794236Z 0 [Note] InnoDB: Using Linux native AIO 2019-01-03T07:38:33.794841Z 0 [Note] InnoDB: Number of pools: 1 2019-01-03T07:38:33.795106Z 0 [Note] InnoDB: Using CPU crc32 instructions 2019-01-03T07:38:33.803488Z 0 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M 2019-01-03T07:38:33.816970Z 0 [Note] InnoDB: Completed initialization of buffer pool 2019-01-03T07:38:33.822663Z 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2019-01-03T07:38:34.030135Z 0 [Note] InnoDB: Crash recovery did not find the parallel doublewrite buffer at /mysql/data/xb_doublewrite 2019-01-03T07:38:34.106981Z 0 [Note] InnoDB: Highest supported file format is Barracuda. 2019-01-03T07:38:34.177702Z 0 [Note] InnoDB: Created parallel doublewrite buffer at /mysql/data/xb_doublewrite, size 3932160 bytes 2019-01-03T07:38:34.789139Z 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2019-01-03T07:38:34.789317Z 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2019-01-03T07:38:34.829979Z 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2019-01-03T07:38:34.833636Z 0 [Note] InnoDB: 96 redo rollback segment(s) found. 96 redo rollback segment(s) are active. 2019-01-03T07:38:34.833659Z 0 [Note] InnoDB: 32 non-redo rollback segment(s) are active. 2019-01-03T07:38:34.834379Z 0 [Note] InnoDB: Waiting for purge to start 2019-01-03T07:38:34.884549Z 0 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.7.23-23 started; log sequence number 2594863 2019-01-03T07:38:34.884599Z 0 [Warning] InnoDB: Skipping buffer pool dump/restore during wsrep recovery. 2019-01-03T07:38:34.884934Z 0 [Note] Plugin 'FEDERATED' is disabled. 2019-01-03T07:38:34.913724Z 0 [Note] WSREP: Recovered position: 00000000-0000-0000-0000-000000000000:-1 2019-01-03T07:38:34.913764Z 0 [Note] Binlog end 2019-01-03T07:38:34.913870Z 0 [Note] Shutting down plugin 'ngram' 2019-01-03T07:38:34.913890Z 0 [Note] Shutting down plugin 'partition' 2019-01-03T07:38:34.913896Z 0 [Note] Shutting down plugin 'ARCHIVE' 2019-01-03T07:38:34.913901Z 0 [Note] Shutting down plugin 'BLACKHOLE' 2019-01-03T07:38:34.913907Z 0 [Note] Shutting down plugin 'INNODB_SYS_VIRTUAL' 2019-01-03T07:38:34.913913Z 0 [Note] Shutting down plugin 'INNODB_CHANGED_PAGES' 2019-01-03T07:38:34.913917Z 0 [Note] Shutting down plugin 'INNODB_SYS_DATAFILES' 2019-01-03T07:38:34.913922Z 0 [Note] Shutting down plugin 'INNODB_SYS_TABLESPACES' 2019-01-03T07:38:34.913927Z 0 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN_COLS' 2019-01-03T07:38:34.913932Z 0 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN' 2019-01-03T07:38:34.913936Z 0 [Note] Shutting down plugin 'INNODB_SYS_FIELDS' 2019-01-03T07:38:34.913941Z 0 [Note] Shutting down plugin 'INNODB_SYS_COLUMNS' 2019-01-03T07:38:34.913946Z 0 [Note] Shutting down plugin 'INNODB_SYS_INDEXES' 2019-01-03T07:38:34.913950Z 0 [Note] Shutting down plugin 'INNODB_SYS_TABLESTATS' 2019-01-03T07:38:34.913955Z 0 [Note] Shutting down plugin 'INNODB_SYS_TABLES' 2019-01-03T07:38:34.913960Z 0 [Note] Shutting down plugin 'INNODB_FT_INDEX_TABLE' 2019-01-03T07:38:34.913964Z 0 [Note] Shutting down plugin 'INNODB_FT_INDEX_CACHE' 2019-01-03T07:38:34.913969Z 0 [Note] Shutting down plugin 'INNODB_FT_CONFIG' 2019-01-03T07:38:34.913973Z 0 [Note] Shutting down plugin 'INNODB_FT_BEING_DELETED' 2019-01-03T07:38:34.913978Z 0 [Note] Shutting down plugin 'INNODB_FT_DELETED' 2019-01-03T07:38:34.913983Z 0 [Note] Shutting down plugin 'INNODB_FT_DEFAULT_STOPWORD' 2019-01-03T07:38:34.913995Z 0 [Note] Shutting down plugin 'INNODB_METRICS' 2019-01-03T07:38:34.914000Z 0 [Note] Shutting down plugin 'INNODB_TEMP_TABLE_INFO' 2019-01-03T07:38:34.914005Z 0 [Note] Shutting down plugin 'INNODB_BUFFER_POOL_STATS' 2019-01-03T07:38:34.914010Z 0 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE_LRU' 2019-01-03T07:38:34.914014Z 0 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE' 2019-01-03T07:38:34.914019Z 0 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX_RESET' 2019-01-03T07:38:34.914024Z 0 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX' 2019-01-03T07:38:34.914028Z 0 [Note] Shutting down plugin 'INNODB_CMPMEM_RESET' 2019-01-03T07:38:34.914033Z 0 [Note] Shutting down plugin 'INNODB_CMPMEM' 2019-01-03T07:38:34.914038Z 0 [Note] Shutting down plugin 'INNODB_CMP_RESET' 2019-01-03T07:38:34.914042Z 0 [Note] Shutting down plugin 'INNODB_CMP' 2019-01-03T07:38:34.914047Z 0 [Note] Shutting down plugin 'INNODB_LOCK_WAITS' 2019-01-03T07:38:34.914052Z 0 [Note] Shutting down plugin 'INNODB_LOCKS' 2019-01-03T07:38:34.914056Z 0 [Note] Shutting down plugin 'INNODB_TRX' 2019-01-03T07:38:34.914061Z 0 [Note] Shutting down plugin 'XTRADB_ZIP_DICT_COLS' 2019-01-03T07:38:34.914066Z 0 [Note] Shutting down plugin 'XTRADB_ZIP_DICT' 2019-01-03T07:38:34.914070Z 0 [Note] Shutting down plugin 'XTRADB_RSEG' 2019-01-03T07:38:34.914075Z 0 [Note] Shutting down plugin 'XTRADB_INTERNAL_HASH_TABLES' 2019-01-03T07:38:34.914080Z 0 [Note] Shutting down plugin 'XTRADB_READ_VIEW' 2019-01-03T07:38:34.914085Z 0 [Note] Shutting down plugin 'InnoDB' 2019-01-03T07:38:34.914199Z 0 [Note] InnoDB: FTS optimize thread exiting. 2019-01-03T07:38:34.914309Z 0 [Note] InnoDB: Starting shutdown... 2019-01-03T07:38:36.015589Z 0 [Note] InnoDB: Waiting for page_cleaner to finish flushing of buffer pool 2019-01-03T07:38:37.535592Z 0 [Note] InnoDB: Shutdown completed; log sequence number 2594882 2019-01-03T07:38:37.539061Z 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2019-01-03T07:38:37.539083Z 0 [Note] Shutting down plugin 'MyISAM' 2019-01-03T07:38:37.539493Z 0 [Note] Shutting down plugin 'MRG_MYISAM' 2019-01-03T07:38:37.539527Z 0 [Note] Shutting down plugin 'CSV' 2019-01-03T07:38:37.539539Z 0 [Note] Shutting down plugin 'MEMORY' 2019-01-03T07:38:37.539552Z 0 [Note] Shutting down plugin 'PERFORMANCE_SCHEMA' 2019-01-03T07:38:37.539636Z 0 [Note] Shutting down plugin 'sha256_password' 2019-01-03T07:38:37.539642Z 0 [Note] Shutting down plugin 'mysql_native_password' 2019-01-03T07:38:37.539647Z 0 [Note] Shutting down plugin 'wsrep' 2019-01-03T07:38:37.539867Z 0 [Note] Shutting down plugin 'binlog' 2019-01-03T07:38:37.540134Z 0 [Note] /usr/sbin/mysqld: Shutdown complete 2019-01-03T07:38:37.847288Z 0 [Note] /usr/sbin/mysqld (mysqld 5.7.23-23-57-log) starting as process 121854 ... 2019-01-03T07:38:37.857722Z 0 [Note] WSREP: Setting wsrep_ready to false 2019-01-03T07:38:37.857779Z 0 [Note] WSREP: No pre-stored wsrep-start position found. Skipping position initialization. 2019-01-03T07:38:37.857788Z 0 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib64/galera3/libgalera_smm.so' 2019-01-03T07:38:38.221234Z 0 [Note] WSREP: wsrep_load(): Galera 3.31(rf216443) by Codership Oy loaded successfully. 2019-01-03T07:38:38.221349Z 0 [Note] WSREP: CRC-32C: using hardware acceleration. 2019-01-03T07:38:38.254414Z 0 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1, safe_to_bootstrap: 1 2019-01-03T07:38:38.259379Z 0 [Note] WSREP: Passing config to GCS: base_dir = /mysql/data/; base_host = 172.19.101.108; base_port = 4567; cert.log_conflicts = no; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 10; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 4; evs.view_forget_timeout = PT24H; gcache.dir = /mysql/data/; gcache.freeze_purge_at_seqno = -1; gcache.keep_pages_count = 0; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /mysql/data//galera.cache; gcache.page_size = 128M; gcache.recover = no; gcache.size = 128M; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1; gcs.fc_limit = 100; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc.checksum = false; pc.ignore_quorum = false; pc.ignore_sb = false; pc.npvo = false; pc.recovery = 1; pc.version = 0; pc.wait_prim = true; pc.wait_prim_timeout = PT30S; pc.weight = 1; protonet.backend = asio; protonet.version = 0; repl.causal_read_timeout = PT30S; repl.commit_order = 3; repl.key_format = FLAT8; repl.max_ws_size = 2147483647; repl.proto_max = 9; socket.checksum = 2; socket.recv_buf_size = 212992; 2019-01-03T07:38:38.281517Z 0 [Note] WSREP: GCache history reset: 00000000-0000-0000-0000-000000000000:0 -> 00000000-0000-0000-0000-000000000000:-1 2019-01-03T07:38:38.282279Z 0 [Note] WSREP: Assign initial position for certification: -1, protocol version: -1 2019-01-03T07:38:38.282312Z 0 [Note] WSREP: Preparing to initiate SST/IST 2019-01-03T07:38:38.282321Z 0 [Note] WSREP: Starting replication 2019-01-03T07:38:38.282340Z 0 [Note] WSREP: Setting initial position to 00000000-0000-0000-0000-000000000000:-1 2019-01-03T07:38:38.290275Z 0 [Note] WSREP: Using CRC-32C for message checksums. 2019-01-03T07:38:38.290433Z 0 [Note] WSREP: gcomm thread scheduling priority set to other:0 2019-01-03T07:38:38.291163Z 0 [Warning] WSREP: Fail to access the file (/mysql/data//gvwstate.dat) error (No such file or directory). It is possible if node is booting for first time or re-booting after a graceful shutdown 2019-01-03T07:38:38.291182Z 0 [Note] WSREP: Restoring primary-component from disk failed. Either node is booting for first time or re-booting after a graceful shutdown 2019-01-03T07:38:38.291982Z 0 [Note] WSREP: GMCast version 0 2019-01-03T07:38:38.292212Z 0 [Note] WSREP: (95828265, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567 2019-01-03T07:38:38.292227Z 0 [Note] WSREP: (95828265, 'tcp://0.0.0.0:4567') multicast: , ttl: 1 2019-01-03T07:38:38.292600Z 0 [Note] WSREP: EVS version 0 2019-01-03T07:38:38.292727Z 0 [Note] WSREP: gcomm: connecting to group 'fmk113114', peer '172.19.101.107:,172.19.101.108:' 2019-01-03T07:38:38.295856Z 0 [Note] WSREP: (95828265, 'tcp://0.0.0.0:4567') connection established to 95828265 tcp://172.19.101.108:4567 2019-01-03T07:38:38.295898Z 0 [Warning] WSREP: (95828265, 'tcp://0.0.0.0:4567') address 'tcp://172.19.101.108:4567' points to own listening address, blacklisting 2019-01-03T07:38:38.567242Z 0 [Note] WSREP: (95828265, 'tcp://0.0.0.0:4567') connection established to 9cca12e4 tcp://172.19.101.107:4567 2019-01-03T07:38:38.567371Z 0 [Note] WSREP: (95828265, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: 2019-01-03T07:38:38.821808Z 0 [Note] WSREP: declaring 9cca12e4 at tcp://172.19.101.107:4567 stable 2019-01-03T07:38:38.822622Z 0 [Note] WSREP: Node 9cca12e4 state primary 2019-01-03T07:38:38.832608Z 0 [Note] WSREP: Current view of cluster as seen by this node view (view_id(PRIM,95828265,14) memb { 95828265,0 9cca12e4,0 } joined { } left { } partitioned { } ) 2019-01-03T07:38:38.832641Z 0 [Note] WSREP: Save the discovered primary-component to disk 2019-01-03T07:38:39.295072Z 0 [Note] WSREP: gcomm: connected 2019-01-03T07:38:39.295362Z 0 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0) 2019-01-03T07:38:39.295465Z 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 2 2019-01-03T07:38:39.295573Z 0 [Note] WSREP: Waiting for SST/IST to complete. 2019-01-03T07:38:39.296288Z 0 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 961bc131-0f2a-11e9-87e4-46f353590347 2019-01-03T07:38:39.296810Z 0 [Note] WSREP: STATE EXCHANGE: sent state msg: 961bc131-0f2a-11e9-87e4-46f353590347 2019-01-03T07:38:39.297080Z 0 [Note] WSREP: STATE EXCHANGE: got state msg: 961bc131-0f2a-11e9-87e4-46f353590347 from 0 (pld1fmk0114) 2019-01-03T07:38:39.297358Z 0 [Note] WSREP: STATE EXCHANGE: got state msg: 961bc131-0f2a-11e9-87e4-46f353590347 from 1 (pld1fmk0113) 2019-01-03T07:38:39.297373Z 0 [Note] WSREP: Quorum results: version = 4, component = PRIMARY, conf_id = 13, members = 1/2 (primary/total), act_id = 0, last_appl. = -1, protocols = 0/9/3 (gcs/repl/appl), group UUID = 9ccc59d2-04c4-11e9-81f3-a650bf4e5391 2019-01-03T07:38:39.297386Z 0 [Note] WSREP: Flow-control interval: [141, 141] 2019-01-03T07:38:39.297393Z 0 [Note] WSREP: Trying to continue unpaused monitor 2019-01-03T07:38:39.297399Z 0 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 0) 2019-01-03T07:38:39.297488Z 1 [Note] WSREP: State transfer required: Group state: 9ccc59d2-04c4-11e9-81f3-a650bf4e5391:0 Local state: 00000000-0000-0000-0000-000000000000:-1 2019-01-03T07:38:39.297520Z 1 [Note] WSREP: REPL Protocols: 9 (4, 2) 2019-01-03T07:38:39.297546Z 1 [Note] WSREP: New cluster view: global state: 9ccc59d2-04c4-11e9-81f3-a650bf4e5391:0, view# 14: Primary, number of nodes: 2, my index: 0, protocol version 3 2019-01-03T07:38:39.297557Z 1 [Note] WSREP: Setting wsrep_ready to true 2019-01-03T07:38:39.297563Z 1 [Warning] WSREP: Gap in state sequence. Need state transfer. 2019-01-03T07:38:39.297569Z 1 [Note] WSREP: Setting wsrep_ready to false 2019-01-03T07:38:39.297662Z 0 [Note] WSREP: Initiating SST/IST transfer on JOINER side (wsrep_sst_xtrabackup-v2 --role 'joiner' --address '172.19.101.107' --datadir '/mysql/data/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --parent '121854' --mysqld-version '5.7.23-23-57' '' ) 2019-01-03T07:38:40.428391Z WSREP_SST: [WARNING] wsrep_node_address or wsrep_sst_receive_address not set. Consider setting them if SST fails. 2019-01-03T07:38:40.608552Z WSREP_SST: [WARNING] Found a stale sst_in_progress file: /mysql/data//sst_in_progress 2019-01-03T07:38:41.017278Z 1 [Note] WSREP: Prepared SST/IST request: xtrabackup-v2|172.19.101.107:4444/xtrabackup_sst//1 2019-01-03T07:38:41.017310Z 1 [Note] WSREP: Auto Increment Offset/Increment re-align with cluster membership change (Offset: 1 -> 1) (Increment: 1 -> 2) 2019-01-03T07:38:41.017324Z 1 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 2019-01-03T07:38:41.017361Z 1 [Note] WSREP: Assign initial position for certification: 0, protocol version: 4 2019-01-03T07:38:41.017415Z 0 [Note] WSREP: Service thread queue flushed. 2019-01-03T07:38:41.017451Z 1 [Note] WSREP: Check if state gap can be serviced using IST 2019-01-03T07:38:41.017469Z 1 [Note] WSREP: Local UUID: 00000000-0000-0000-0000-000000000000 != Group UUID: 9ccc59d2-04c4-11e9-81f3-a650bf4e5391 2019-01-03T07:38:41.017534Z 1 [Note] WSREP: State gap can't be serviced using IST. Switching to SST 2019-01-03T07:38:41.017544Z 1 [Note] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (9ccc59d2-04c4-11e9-81f3-a650bf4e5391): 1 (Operation not permitted) at galera/src/replicator_str.cpp:prepare_for_IST():538. IST will be unavailable. 2019-01-03T07:38:41.018040Z 0 [Warning] WSREP: Member 0.0 (pld1fmk0114) requested state transfer from '172.19.101.107', but it is impossible to select State Transfer donor: No route to host 2019-01-03T07:38:41.018106Z 1 [ERROR] WSREP: Requesting state transfer failed: -113(No route to host) 2019-01-03T07:38:41.018136Z 1 [ERROR] WSREP: State transfer request failed unrecoverably: 113 (No route to host). Most likely it is due to inability to communicate with the cluster primary component. Restart required. 2019-01-03T07:38:41.018147Z 1 [Note] WSREP: Closing send monitor... 2019-01-03T07:38:41.018154Z 1 [Note] WSREP: Closed send monitor. 2019-01-03T07:38:41.018165Z 1 [Note] WSREP: gcomm: terminating thread 2019-01-03T07:38:41.018181Z 1 [Note] WSREP: gcomm: joining thread 2019-01-03T07:38:41.018249Z 1 [Note] WSREP: gcomm: closing backend 2019-01-03T07:38:41.795505Z 1 [Note] WSREP: (95828265, 'tcp://0.0.0.0:4567') turning message relay requesting off 2019-01-03T07:38:44.295403Z 1 [Note] WSREP: (95828265, 'tcp://0.0.0.0:4567') connection to peer 9cca12e4 with addr tcp://172.19.101.107:4567 timed out, no messages seen in PT3S (gmcast.peer_timeout) 2019-01-03T07:38:44.295573Z 1 [Note] WSREP: (95828265, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://172.19.101.107:4567 2019-01-03T07:38:45.795374Z 1 [Note] WSREP: (95828265, 'tcp://0.0.0.0:4567') reconnecting to 9cca12e4 (tcp://172.19.101.107:4567), attempt 0 2019-01-03T07:38:46.518518Z 1 [Note] WSREP: declaring node with index 1 suspected, timeout PT5S (evs.suspect_timeout) 2019-01-03T07:38:46.518594Z 1 [Note] WSREP: evs::proto(95828265, LEAVING, view_id(REG,95828265,14)) suspecting node: 9cca12e4 2019-01-03T07:38:46.518606Z 1 [Note] WSREP: evs::proto(95828265, LEAVING, view_id(REG,95828265,14)) suspected node without join message, declaring inactive 2019-01-03T07:38:46.518650Z 1 [Note] WSREP: Current view of cluster as seen by this node view (view_id(NON_PRIM,95828265,14) memb { 95828265,0 } joined { } left { } partitioned { 9cca12e4,0 } ) 2019-01-03T07:38:46.518723Z 1 [Note] WSREP: Current view of cluster as seen by this node view ((empty)) 2019-01-03T07:38:46.519062Z 1 [Note] WSREP: gcomm: closed 2019-01-03T07:38:46.519102Z 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1 2019-01-03T07:38:46.519140Z 0 [Note] WSREP: Flow-control interval: [100, 100] 2019-01-03T07:38:46.519149Z 0 [Note] WSREP: Trying to continue unpaused monitor 2019-01-03T07:38:46.519155Z 0 [Note] WSREP: Received NON-PRIMARY. 2019-01-03T07:38:46.519161Z 0 [Note] WSREP: Shifting PRIMARY -> OPEN (TO: 0) 2019-01-03T07:38:46.519172Z 0 [Note] WSREP: Received self-leave message. 2019-01-03T07:38:46.519180Z 0 [Note] WSREP: Flow-control interval: [0, 0] 2019-01-03T07:38:46.519185Z 0 [Note] WSREP: Trying to continue unpaused monitor 2019-01-03T07:38:46.519190Z 0 [Note] WSREP: Received SELF-LEAVE. Closing connection. 2019-01-03T07:38:46.519196Z 0 [Note] WSREP: Shifting OPEN -> CLOSED (TO: 0) 2019-01-03T07:38:46.519206Z 0 [Note] WSREP: RECV thread exiting 0: Success 2019-01-03T07:38:46.519251Z 1 [Note] WSREP: recv_thread() joined. 2019-01-03T07:38:46.519261Z 1 [Note] WSREP: Closing replication queue. 2019-01-03T07:38:46.519267Z 1 [Note] WSREP: Closing slave action queue. 2019-01-03T07:38:46.519282Z 1 [Note] WSREP: /usr/sbin/mysqld: Terminated. 2019-01-03T07:40:20.692631Z WSREP_SST: [ERROR] ******************* FATAL ERROR ********************** 2019-01-03T07:40:20.711208Z WSREP_SST: [ERROR] Error while getting data from donor node: exit codes: 137 0 2019-01-03T07:40:20.717010Z WSREP_SST: [ERROR] ****************************************************** 2019-01-03T07:40:20.722328Z WSREP_SST: [ERROR] Cleanup after exit with status:32 [root@pld1fmk0114 ~]#