Hi, i have cluster of 3 nodes, for some reasons the the infrastructure went done, and it has been done for some time. now i’m attempting to bring back the cluster. 1 node have been bootstrapped but reconnecting other two node has been a challenge despite of googling and tried proposed solutions. attached are logs for both donor and joiner. Both node Selinux is disabled and ports allowed like 4444
Donor logs
2021-11-10T17:59:32.307847Z 0 [Note] WSREP: cleaning up e4d1b4a9 (tcp://10.0.0.8:4567)
2021-11-11T07:23:34.076583Z 359850 [Note] Aborted connection 359850 to db: 'municipal_manica_prod' user: 'chimoioAppUser' host: 'db-lb-02' (Got an error reading communication packets)
2021-11-11T07:51:36.898434Z 363813 [Note] Aborted connection 363813 to db: 'municipal_manica_prod' user: 'chimoioAppUser' host: 'db-lb-02' (Got an error reading communication packets)
2021-11-11T15:32:24.845179Z 0 [Note] WSREP: (df8a32a7, 'tcp://0.0.0.0:4567') connection established to 86967d00 tcp://10.0.0.8:4567
2021-11-11T15:32:24.845612Z 0 [Note] WSREP: (df8a32a7, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers:
2021-11-11T15:32:24.993867Z 0 [Note] WSREP: declaring 86967d00 at tcp://10.0.0.8:4567 stable
2021-11-11T15:32:24.994336Z 0 [Note] WSREP: Node df8a32a7 state primary
2021-11-11T15:32:24.995664Z 0 [Note] WSREP: Current view of cluster as seen by this node
view (view_id(PRIM,86967d00,62)
memb {
86967d00,0
df8a32a7,0
}
joined {
}
left {
}
partitioned {
}
)
2021-11-11T15:32:24.995689Z 0 [Note] WSREP: Save the discovered primary-component to disk
2021-11-11T15:32:24.997268Z 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 2
2021-11-11T15:32:24.997286Z 0 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
2021-11-11T15:32:25.344713Z 0 [Note] WSREP: STATE EXCHANGE: sent state msg: 86e30ed1-4304-11ec-a42f-ee052945f6fe
2021-11-11T15:32:25.344873Z 0 [Note] WSREP: STATE EXCHANGE: got state msg: 86e30ed1-4304-11ec-a42f-ee052945f6fe from 0 (db-server-01)
2021-11-11T15:32:25.345068Z 0 [Note] WSREP: STATE EXCHANGE: got state msg: 86e30ed1-4304-11ec-a42f-ee052945f6fe from 1 (db-server-02)
2021-11-11T15:32:25.345082Z 0 [Note] WSREP: Quorum results:
version = 6,
component = PRIMARY,
conf_id = 61,
members = 1/2 (primary/total),
act_id = 211395,
last_appl. = 211361,
protocols = 0/9/3 (gcs/repl/appl),
group UUID = 7fdc5f55-4073-11ec-b2d6-8606e13ac974
2021-11-11T15:32:25.345092Z 0 [Note] WSREP: Flow-control interval: [141, 141]
2021-11-11T15:32:25.345098Z 0 [Note] WSREP: Trying to continue unpaused monitor
2021-11-11T15:32:25.345284Z 6 [Note] WSREP: REPL Protocols: 9 (4, 2)
2021-11-11T15:32:25.345356Z 6 [Note] WSREP: New cluster view: global state: 7fdc5f55-4073-11ec-b2d6-8606e13ac974:211395, view# 62: Primary, number of nodes: 2, my index: 1, protocol version 3
2021-11-11T15:32:25.345388Z 6 [Note] WSREP: Setting wsrep_ready to true
2021-11-11T15:32:25.345394Z 6 [Note] WSREP: Auto Increment Offset/Increment re-align with cluster membership change (Offset: 1 -> 2) (Increment: 1 -> 2)
2021-11-11T15:32:25.345399Z 6 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2021-11-11T15:32:25.345493Z 6 [Note] WSREP: Assign initial position for certification: 211395, protocol version: 4
2021-11-11T15:32:25.345520Z 0 [Note] WSREP: Service thread queue flushed.
2021-11-11T15:32:25.913882Z 0 [Note] WSREP: Member 0.0 (db-server-01) requested state transfer from '*any*'. Selected 1.0 (db-server-02)(SYNCED) as donor.
2021-11-11T15:32:25.913919Z 0 [Note] WSREP: Shifting SYNCED -> DONOR/DESYNCED (TO: 211412)
2021-11-11T15:32:25.913948Z 6 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2021-11-11T15:32:25.914749Z 0 [Note] WSREP: Initiating SST/IST transfer on DONOR side (wsrep_sst_xtrabackup-v2 --role 'donor' --address '10.0.0.8:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --mysqld-version '5.7.28-31-57' '' --gtid '7fdc5f55-4073-11ec-b2d6-8606e13ac974:211412' )
2021-11-11T15:32:25.915178Z 6 [Note] WSREP: DONOR thread signaled with 0
2021-11-11T15:32:25.915568Z 0 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'donor' --address '10.0.0.8:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --mysqld-version '5.7.28-31-57' '' --gtid '7fdc5f55-4073-11ec-b2d6-8606e13ac974:211412' : 2 (No such file or directory)
2021-11-11T15:32:25.915638Z 0 [ERROR] WSREP: Command did not run: wsrep_sst_xtrabackup-v2 --role 'donor' --address '10.0.0.8:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --mysqld-version '5.7.28-31-57' '' --gtid '7fdc5f55-4073-11ec-b2d6-8606e13ac974:211412'
2021-11-11T15:32:25.915937Z 0 [Warning] WSREP: 1.0 (db-server-02): State transfer to 0.0 (db-server-01) failed: -2 (No such file or directory)
2021-11-11T15:32:25.915947Z 0 [Note] WSREP: Shifting DONOR/DESYNCED -> JOINED (TO: 211412)
2021-11-11T15:32:25.916231Z 0 [Note] WSREP: Member 1.0 (db-server-02) synced with group.
2021-11-11T15:32:25.916244Z 0 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 211412)
2021-11-11T15:32:25.916271Z 10 [Note] WSREP: Synchronized with group, ready for connections
2021-11-11T15:32:25.916282Z 10 [Note] WSREP: Setting wsrep_ready to true
2021-11-11T15:32:25.916288Z 10 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2021-11-11T15:32:26.916552Z 0 [Note] WSREP: forgetting 86967d00 (tcp://10.0.0.8:4567)
2021-11-11T15:32:26.916617Z 0 [Note] WSREP: Node df8a32a7 state primary
2021-11-11T15:32:26.916640Z 0 [Note] WSREP: Current view of cluster as seen by this node
view (view_id(PRIM,df8a32a7,63)
memb {
df8a32a7,0
}
joined {
}
left {
}
partitioned {
86967d00,0
}
)
2021-11-11T15:32:26.916651Z 0 [Note] WSREP: Save the discovered primary-component to disk
2021-11-11T15:32:26.917811Z 0 [Note] WSREP: forgetting 86967d00 (tcp://10.0.0.8:4567)
2021-11-11T15:32:26.917982Z 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 1
2021-11-11T15:32:26.918060Z 0 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 9323f42c-4304-11ec-b088-da49c3b32a30
2021-11-11T15:32:26.918076Z 0 [Note] WSREP: STATE EXCHANGE: sent state msg: 9323f42c-4304-11ec-b088-da49c3b32a30
2021-11-11T15:32:26.918082Z 0 [Note] WSREP: STATE EXCHANGE: got state msg: 9323f42c-4304-11ec-b088-da49c3b32a30 from 0 (db-server-02)
2021-11-11T15:32:26.918090Z 0 [Note] WSREP: Quorum results:
version = 6,
component = PRIMARY,
conf_id = 62,
members = 1/1 (primary/total),
act_id = 211412,
last_appl. = 211361,
protocols = 0/9/3 (gcs/repl/appl),
group UUID = 7fdc5f55-4073-11ec-b2d6-8606e13ac974
2021-11-11T15:32:26.918096Z 0 [Note] WSREP: Flow-control interval: [100, 100]
2021-11-11T15:32:26.918099Z 0 [Note] WSREP: Trying to continue unpaused monitor
2021-11-11T15:32:26.918394Z 3 [Note] WSREP: REPL Protocols: 9 (4, 2)
2021-11-11T15:32:26.918417Z 3 [Note] WSREP: New cluster view: global state: 7fdc5f55-4073-11ec-b2d6-8606e13ac974:211412, view# 63: Primary, number of nodes: 1, my index: 0, protocol version 3
2021-11-11T15:32:26.918440Z 3 [Note] WSREP: Setting wsrep_ready to true
2021-11-11T15:32:26.918446Z 3 [Note] WSREP: Auto Increment Offset/Increment re-align with cluster membership change (Offset: 2 -> 1) (Increment: 2 -> 1)
2021-11-11T15:32:26.918487Z 3 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2021-11-11T15:32:26.918547Z 3 [Note] WSREP: Assign initial position for certification: 211412, protocol version: 4
2021-11-11T15:32:26.918586Z 0 [Note] WSREP: Service thread queue flushed.
2021-11-11T15:32:28.338792Z 0 [Note] WSREP: (df8a32a7, 'tcp://0.0.0.0:4567') turning message relay requesting off
2021-11-11T15:32:32.339481Z 0 [Note] WSREP: cleaning up 86967d00 (tcp://10.0.0.8:4567)
Joiner logs
Log of wsrep recovery (--wsrep-recover):
2021-11-11T15:32:02.855734Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2021-11-11T15:32:02.857365Z 0 [Note] /usr/sbin/mysqld (mysqld 5.7.28-31-57-log) starting as process 7105 ...
2021-11-11T15:32:02.861620Z 0 [Warning] No argument was provided to --log-bin, and --log-bin-index was not used; so replication may break when this MySQL server acts as a master and has his hostname changed!! Please use '--log-bin=db-server-01-bin' to avoid this problem.
2021-11-11T15:32:02.864028Z 0 [Note] InnoDB: PUNCH HOLE support available
2021-11-11T15:32:02.864051Z 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2021-11-11T15:32:02.864055Z 0 [Note] InnoDB: Uses event mutexes
2021-11-11T15:32:02.864059Z 0 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2021-11-11T15:32:02.864062Z 0 [Note] InnoDB: Compressed tables use zlib 1.2.7
2021-11-11T15:32:02.864068Z 0 [Note] InnoDB: Using Linux native AIO
2021-11-11T15:32:02.864710Z 0 [Note] InnoDB: Number of pools: 1
2021-11-11T15:32:02.864931Z 0 [Note] InnoDB: Using CPU crc32 instructions
2021-11-11T15:32:02.867165Z 0 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M
2021-11-11T15:32:02.872467Z 0 [Note] InnoDB: Completed initialization of buffer pool
2021-11-11T15:32:02.876339Z 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2021-11-11T15:32:02.889685Z 0 [Note] InnoDB: Crash recovery did not find the parallel doublewrite buffer at /var/lib/mysql/xb_doublewrite
2021-11-11T15:32:02.890459Z 0 [Note] InnoDB: Highest supported file format is Barracuda.
2021-11-11T15:32:02.933116Z 0 [Note] InnoDB: Created parallel doublewrite buffer at /var/lib/mysql/xb_doublewrite, size 3932160 bytes
2021-11-11T15:32:02.939306Z 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2021-11-11T15:32:02.939366Z 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2021-11-11T15:32:03.003077Z 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2021-11-11T15:32:03.003962Z 0 [Note] InnoDB: 96 redo rollback segment(s) found. 96 redo rollback segment(s) are active.
2021-11-11T15:32:03.003979Z 0 [Note] InnoDB: 32 non-redo rollback segment(s) are active.
2021-11-11T15:32:03.004942Z 0 [Note] InnoDB: Waiting for purge to start
2021-11-11T15:32:03.055069Z 0 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.7.28-31 started; log sequence number 2634534
2021-11-11T15:32:03.055116Z 0 [Warning] InnoDB: Skipping buffer pool dump/restore during wsrep recovery.
2021-11-11T15:32:03.055516Z 0 [Note] Plugin 'FEDERATED' is disabled.
2021-11-11T15:32:03.061369Z 0 [Note] WSREP: Recovered position: 00000000-0000-0000-0000-000000000000:-1
2021-11-11T15:32:03.061393Z 0 [Note] Binlog end
2021-11-11T15:32:03.061474Z 0 [Note] Shutting down plugin 'ngram'
2021-11-11T15:32:03.061483Z 0 [Note] Shutting down plugin 'BLACKHOLE'
2021-11-11T15:32:03.061487Z 0 [Note] Shutting down plugin 'ARCHIVE'
2021-11-11T15:32:03.061490Z 0 [Note] Shutting down plugin 'partition'
2021-11-11T15:32:03.061492Z 0 [Note] Shutting down plugin 'MRG_MYISAM'
2021-11-11T15:32:03.061495Z 0 [Note] Shutting down plugin 'MyISAM'
2021-11-11T15:32:03.061513Z 0 [Note] Shutting down plugin 'CSV'
2021-11-11T15:32:03.061516Z 0 [Note] Shutting down plugin 'MEMORY'
2021-11-11T15:32:03.061526Z 0 [Note] Shutting down plugin 'INNODB_TABLESPACES_SCRUBBING'
2021-11-11T15:32:03.061529Z 0 [Note] Shutting down plugin 'INNODB_TABLESPACES_ENCRYPTION'
2021-11-11T15:32:03.061532Z 0 [Note] Shutting down plugin 'INNODB_SYS_VIRTUAL'
2021-11-11T15:32:03.061534Z 0 [Note] Shutting down plugin 'INNODB_CHANGED_PAGES'
2021-11-11T15:32:03.061537Z 0 [Note] Shutting down plugin 'INNODB_SYS_DATAFILES'
2021-11-11T15:32:03.061539Z 0 [Note] Shutting down plugin 'INNODB_SYS_TABLESPACES'
2021-11-11T15:32:03.061541Z 0 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN_COLS'
2021-11-11T15:32:03.061562Z 0 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN'
2021-11-11T15:32:03.061568Z 0 [Note] Shutting down plugin 'INNODB_SYS_FIELDS'
2021-11-11T15:32:03.061570Z 0 [Note] Shutting down plugin 'INNODB_SYS_COLUMNS'
2021-11-11T15:32:03.061573Z 0 [Note] Shutting down plugin 'INNODB_SYS_INDEXES'
2021-11-11T15:32:03.061575Z 0 [Note] Shutting down plugin 'INNODB_SYS_TABLESTATS'
2021-11-11T15:32:03.061577Z 0 [Note] Shutting down plugin 'INNODB_SYS_TABLES'
2021-11-11T15:32:03.061579Z 0 [Note] Shutting down plugin 'INNODB_FT_INDEX_TABLE'
2021-11-11T15:32:03.061582Z 0 [Note] Shutting down plugin 'INNODB_FT_INDEX_CACHE'
2021-11-11T15:32:03.061584Z 0 [Note] Shutting down plugin 'INNODB_FT_CONFIG'
2021-11-11T15:32:03.061587Z 0 [Note] Shutting down plugin 'INNODB_FT_BEING_DELETED'
2021-11-11T15:32:03.061589Z 0 [Note] Shutting down plugin 'INNODB_FT_DELETED'
2021-11-11T15:32:03.061591Z 0 [Note] Shutting down plugin 'INNODB_FT_DEFAULT_STOPWORD'
2021-11-11T15:32:03.061594Z 0 [Note] Shutting down plugin 'INNODB_METRICS'
2021-11-11T15:32:03.061596Z 0 [Note] Shutting down plugin 'INNODB_TEMP_TABLE_INFO'
2021-11-11T15:32:03.061598Z 0 [Note] Shutting down plugin 'INNODB_BUFFER_POOL_STATS'
2021-11-11T15:32:03.061601Z 0 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE_LRU'
2021-11-11T15:32:03.061603Z 0 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE'
2021-11-11T15:32:03.061605Z 0 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX_RESET'
2021-11-11T15:32:03.061608Z 0 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX'
2021-11-11T15:32:03.061610Z 0 [Note] Shutting down plugin 'INNODB_CMPMEM_RESET'
2021-11-11T15:32:03.061612Z 0 [Note] Shutting down plugin 'INNODB_CMPMEM'
2021-11-11T15:32:03.061615Z 0 [Note] Shutting down plugin 'INNODB_CMP_RESET'
2021-11-11T15:32:03.061617Z 0 [Note] Shutting down plugin 'INNODB_CMP'
2021-11-11T15:32:03.061619Z 0 [Note] Shutting down plugin 'INNODB_LOCK_WAITS'
2021-11-11T15:32:03.061622Z 0 [Note] Shutting down plugin 'INNODB_LOCKS'
2021-11-11T15:32:03.061624Z 0 [Note] Shutting down plugin 'INNODB_TRX'
2021-11-11T15:32:03.061626Z 0 [Note] Shutting down plugin 'XTRADB_ZIP_DICT_COLS'
2021-11-11T15:32:03.061629Z 0 [Note] Shutting down plugin 'XTRADB_ZIP_DICT'
2021-11-11T15:32:03.061631Z 0 [Note] Shutting down plugin 'XTRADB_RSEG'
2021-11-11T15:32:03.061633Z 0 [Note] Shutting down plugin 'XTRADB_INTERNAL_HASH_TABLES'
2021-11-11T15:32:03.061636Z 0 [Note] Shutting down plugin 'XTRADB_READ_VIEW'
2021-11-11T15:32:03.061638Z 0 [Note] Shutting down plugin 'InnoDB'
2021-11-11T15:32:03.061701Z 0 [Note] InnoDB: FTS optimize thread exiting.
2021-11-11T15:32:03.061811Z 0 [Note] InnoDB: Starting shutdown...
2021-11-11T15:32:04.162736Z 0 [Note] InnoDB: Waiting for page_cleaner to finish flushing of buffer pool
2021-11-11T15:32:05.567415Z 0 [Note] InnoDB: Shutdown completed; log sequence number 2634553
2021-11-11T15:32:05.569398Z 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
2021-11-11T15:32:05.569415Z 0 [Note] Shutting down plugin 'PERFORMANCE_SCHEMA'
2021-11-11T15:32:05.569493Z 0 [Note] Shutting down plugin 'sha256_password'
2021-11-11T15:32:05.569497Z 0 [Note] Shutting down plugin 'mysql_native_password'
2021-11-11T15:32:05.569499Z 0 [Note] Shutting down plugin 'wsrep'
2021-11-11T15:32:05.569702Z 0 [Note] Shutting down plugin 'binlog'
2021-11-11T15:32:05.569902Z 0 [Note] /usr/sbin/mysqld: Shutdown complete
2021-11-11T15:32:05.834951Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2021-11-11T15:32:05.837083Z 0 [Note] /usr/sbin/mysqld (mysqld 5.7.28-31-57-log) starting as process 7140 ...
2021-11-11T15:32:05.839458Z 0 [Warning] No argument was provided to --log-bin, and --log-bin-index was not used; so replication may break when this MySQL server acts as a master and has his hostname changed!! Please use '--log-bin=db-server-01-bin' to avoid this problem.
2021-11-11T15:32:05.839683Z 0 [Note] WSREP: Setting wsrep_ready to false
2021-11-11T15:32:05.839698Z 0 [Note] WSREP: No pre-stored wsrep-start position found. Skipping position initialization.
2021-11-11T15:32:05.839707Z 0 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib64/galera3/libgalera_smm.so'
2021-11-11T15:32:05.842195Z 0 [Note] WSREP: wsrep_load(): Galera 3.41(rb3295e6) by Codership Oy <info@codership.com> loaded successfully.
2021-11-11T15:32:05.842257Z 0 [Note] WSREP: CRC-32C: using hardware acceleration.
2021-11-11T15:32:05.842682Z 0 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1, safe_to_bootstrap: 1
2021-11-11T15:32:05.843973Z 0 [Note] WSREP: Passing config to GCS: base_dir = /var/lib/mysql/; base_host = 10.0.0.8; base_port = 4567; cert.log_conflicts = no; cert.optimistic_pa = yes; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 10; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 4; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.freeze_purge_at_seqno = -1; gcache.keep_pages_count = 0; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.recover = no; gcache.size = 128M; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1; gcs.fc_limit = 100; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc.checksum = false; pc.ignore_quorum = false; pc.ignore_sb = false; pc.npvo = false; pc.recovery = true; pc.version = 0; pc.wait_prim = true; pc.wait_prim_timeout = PT30S; pc.weight = 1; protonet.backend = asio; protonet.version = 0; repl.causal_read_timeout = PT30S; repl.commit_order = 3; repl.key_format = FLAT8; repl.max_ws_size = 2147483647; repl.proto_max = 9; socket.checksum = 2; socket.recv_buf_size = 212992;
2021-11-11T15:32:05.856755Z 0 [Note] WSREP: GCache history reset: 7fdc5f55-4073-11ec-b2d6-8606e13ac974:0 -> 00000000-0000-0000-0000-000000000000:-1
2021-11-11T15:32:05.857804Z 0 [Note] WSREP: Assign initial position for certification: -1, protocol version: -1
2021-11-11T15:32:05.857827Z 0 [Note] WSREP: Preparing to initiate SST/IST
2021-11-11T15:32:05.857831Z 0 [Note] WSREP: Starting replication
2021-11-11T15:32:05.857843Z 0 [Note] WSREP: Setting initial position to 00000000-0000-0000-0000-000000000000:-1
2021-11-11T15:32:05.858013Z 0 [Note] WSREP: Using CRC-32C for message checksums.
2021-11-11T15:32:05.858100Z 0 [Note] WSREP: gcomm thread scheduling priority set to other:0
2021-11-11T15:32:05.858193Z 0 [Warning] WSREP: Fail to access the file (/var/lib/mysql//gvwstate.dat) error (No such file or directory). It is possible if node is booting for first time or re-booting after a graceful shutdown
2021-11-11T15:32:05.858201Z 0 [Note] WSREP: Restoring primary-component from disk failed. Either node is booting for first time or re-booting after a graceful shutdown
2021-11-11T15:32:05.858415Z 0 [Note] WSREP: GMCast version 0
2021-11-11T15:32:05.858696Z 0 [Note] WSREP: (86967d00, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567
2021-11-11T15:32:05.858709Z 0 [Note] WSREP: (86967d00, 'tcp://0.0.0.0:4567') multicast: , ttl: 1
2021-11-11T15:32:05.859053Z 0 [Note] WSREP: EVS version 0
2021-11-11T15:32:05.859165Z 0 [Note] WSREP: gcomm: connecting to group 'pxc-cluster', peer '10.0.0.8:,10.0.0.9:,10.0.0.10:'
2021-11-11T15:32:05.860022Z 0 [Note] WSREP: (86967d00, 'tcp://0.0.0.0:4567') Found matching local endpoint for a connection, blacklisting address tcp://10.0.0.8:4567
2021-11-11T15:32:05.861567Z 0 [Note] WSREP: (86967d00, 'tcp://0.0.0.0:4567') connection established to df8a32a7 tcp://10.0.0.9:4567
2021-11-11T15:32:05.861645Z 0 [Note] WSREP: (86967d00, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers:
2021-11-11T15:32:06.009831Z 0 [Note] WSREP: declaring df8a32a7 at tcp://10.0.0.9:4567 stable
2021-11-11T15:32:06.010340Z 0 [Note] WSREP: Node df8a32a7 state primary
2021-11-11T15:32:06.010685Z 0 [Note] WSREP: Current view of cluster as seen by this node
view (view_id(PRIM,86967d00,62)
memb {
86967d00,0
df8a32a7,0
}
joined {
}
left {
}
partitioned {
}
)
2021-11-11T15:32:06.010704Z 0 [Note] WSREP: Save the discovered primary-component to disk
2021-11-11T15:32:06.011681Z 0 [Note] WSREP: discarding pending addr without UUID: tcp://10.0.0.10:4567
2021-11-11T15:32:06.359801Z 0 [Note] WSREP: gcomm: connected
2021-11-11T15:32:06.359936Z 0 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)
2021-11-11T15:32:06.360015Z 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 2
2021-11-11T15:32:06.360042Z 0 [Note] WSREP: Waiting for SST/IST to complete.
2021-11-11T15:32:06.360150Z 0 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 86e30ed1-4304-11ec-a42f-ee052945f6fe
2021-11-11T15:32:06.360666Z 0 [Note] WSREP: STATE EXCHANGE: sent state msg: 86e30ed1-4304-11ec-a42f-ee052945f6fe
2021-11-11T15:32:06.360927Z 0 [Note] WSREP: STATE EXCHANGE: got state msg: 86e30ed1-4304-11ec-a42f-ee052945f6fe from 0 (db-server-01)
2021-11-11T15:32:06.361355Z 0 [Note] WSREP: STATE EXCHANGE: got state msg: 86e30ed1-4304-11ec-a42f-ee052945f6fe from 1 (db-server-02)
2021-11-11T15:32:06.361366Z 0 [Note] WSREP: Quorum results:
version = 6,
component = PRIMARY,
conf_id = 61,
members = 1/2 (primary/total),
act_id = 211395,
last_appl. = -1,
protocols = 0/9/3 (gcs/repl/appl),
group UUID = 7fdc5f55-4073-11ec-b2d6-8606e13ac974
2021-11-11T15:32:06.361375Z 0 [Note] WSREP: Flow-control interval: [141, 141]
2021-11-11T15:32:06.361381Z 0 [Note] WSREP: Trying to continue unpaused monitor
2021-11-11T15:32:06.361385Z 0 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 211395)
2021-11-11T15:32:06.361444Z 2 [Note] WSREP: State transfer required:
Group state: 7fdc5f55-4073-11ec-b2d6-8606e13ac974:211395
Local state: 00000000-0000-0000-0000-000000000000:-1
2021-11-11T15:32:06.361462Z 2 [Note] WSREP: REPL Protocols: 9 (4, 2)
2021-11-11T15:32:06.361473Z 2 [Note] WSREP: New cluster view: global state: 7fdc5f55-4073-11ec-b2d6-8606e13ac974:211395, view# 62: Primary, number of nodes: 2, my index: 0, protocol version 3
2021-11-11T15:32:06.361479Z 2 [Note] WSREP: Setting wsrep_ready to true
2021-11-11T15:32:06.361484Z 2 [Warning] WSREP: Gap in state sequence. Need state transfer.
2021-11-11T15:32:06.361487Z 2 [Note] WSREP: Setting wsrep_ready to false
2021-11-11T15:32:06.361568Z 0 [Note] WSREP: Initiating SST/IST transfer on JOINER side (wsrep_sst_xtrabackup-v2 --role 'joiner' --address '10.0.0.8' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --parent '7140' --mysqld-version '5.7.28-31-57' '' )
2021-11-11T15:32:06.634708Z WSREP_SST: [WARNING] Found a stale sst_in_progress file: /var/lib/mysql//sst_in_progress
2021-11-11T15:32:06.928618Z 2 [Note] WSREP: Prepared SST/IST request: xtrabackup-v2|10.0.0.8:4444/xtrabackup_sst//1
2021-11-11T15:32:06.928639Z 2 [Note] WSREP: Auto Increment Offset/Increment re-align with cluster membership change (Offset: 1 -> 1) (Increment: 1 -> 2)
2021-11-11T15:32:06.928660Z 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2021-11-11T15:32:06.928680Z 2 [Note] WSREP: Assign initial position for certification: 211395, protocol version: 4
2021-11-11T15:32:06.929288Z 0 [Note] WSREP: Service thread queue flushed.
2021-11-11T15:32:06.929331Z 2 [Note] WSREP: Check if state gap can be serviced using IST
2021-11-11T15:32:06.929342Z 2 [Note] WSREP: Local UUID: 00000000-0000-0000-0000-000000000000 != Group UUID: 7fdc5f55-4073-11ec-b2d6-8606e13ac974
2021-11-11T15:32:06.929386Z 2 [Note] WSREP: State gap can't be serviced using IST. Switching to SST
2021-11-11T15:32:06.929392Z 2 [Note] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (7fdc5f55-4073-11ec-b2d6-8606e13ac974): 1 (Operation not permitted)
at galera/src/replicator_str.cpp:prepare_for_IST():538. IST will be unavailable.
2021-11-11T15:32:06.929956Z 0 [Note] WSREP: Member 0.0 (db-server-01) requested state transfer from '*any*'. Selected 1.0 (db-server-02)(SYNCED) as donor.
2021-11-11T15:32:06.929971Z 0 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 211412)
2021-11-11T15:32:06.929999Z 2 [Note] WSREP: Requesting state transfer: success, donor: 1
2021-11-11T15:32:06.930012Z 2 [Note] WSREP: GCache history reset: 00000000-0000-0000-0000-000000000000:0 -> 7fdc5f55-4073-11ec-b2d6-8606e13ac974:211395
2021-11-11T15:32:06.932314Z 0 [Warning] WSREP: 1.0 (db-server-02): State transfer to 0.0 (db-server-01) failed: -2 (No such file or directory)
2021-11-11T15:32:06.932376Z 0 [ERROR] WSREP: gcs/src/gcs_group.cpp:gcs_group_handle_join_msg():806: Will never receive state. Need to abort.
2021-11-11T15:32:06.932391Z 0 [Note] WSREP: gcomm: terminating thread
2021-11-11T15:32:06.932399Z 0 [Note] WSREP: gcomm: joining thread
2021-11-11T15:32:06.932440Z 0 [Note] WSREP: gcomm: closing backend
2021-11-11T15:32:07.934337Z 0 [Note] WSREP: Current view of cluster as seen by this node
view (view_id(NON_PRIM,86967d00,62)
memb {
86967d00,0
}
joined {
}
left {
}
partitioned {
df8a32a7,0
}
)
2021-11-11T15:32:07.934385Z 0 [Note] WSREP: Current view of cluster as seen by this node
view ((empty))
2021-11-11T15:32:07.934583Z 0 [Note] WSREP: gcomm: closed
2021-11-11T15:32:07.934625Z 0 [Note] WSREP: /usr/sbin/mysqld: Terminated.
Terminated
2021-11-11T15:32:12.418219Z WSREP_SST: [ERROR] Removing /var/lib/mysql//xtrabackup_galera_info file due to signal
2021-11-11T15:32:12.420659Z WSREP_SST: [ERROR] Removing file due to signal
2021-11-11T15:32:12.423151Z WSREP_SST: [ERROR] ******************* FATAL ERROR **********************
2021-11-11T15:32:12.424373Z WSREP_SST: [ERROR] Error while getting data from donor node: exit codes: 143 143
2021-11-11T15:32:12.425610Z WSREP_SST: [ERROR] ******************************************************
2021-11-11T15:32:12.426987Z WSREP_SST: [ERROR] Cleanup after exit with status:32
my.cfg
[root@db-server-01 ~]# cat /etc/my.cnf
[mysql]
bind-address=10.0.0.8
server_id=1
#innodb_force_recovery = 1
wsrep_provider=/usr/lib64/galera3/libgalera_smm.so
wsrep_cluster_name=pxc-cluster
wsrep_cluster_address=gcomm://10.0.0.8,10.0.0.9,10.0.0.10
wsrep_node_name=db-server-01
wsrep_node_address=10.0.0.8
wsrep_sst_method=xtrabackup-v2
wsrep_sst_auth="sstuser:mypassword"
pxc_strict_mode=ENFORCING
binlog_format=ROW
default_storage_engine=InnoDB
innodb_autoinc_lock_mode=2
innodb_ft_min_token_size=1
innodb_log_file_size = 512M
#sql-mode="NO_ENGINE_SUBSTITUTION,ERROR_FOR_DIVISION_BY_ZERO"
#sql_mode = "STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION"
explicit_defaults_for_timestamp = 1
# * Fine Tuning
read_rnd_buffer_size = 10M
join_buffer_size = 512M
key_buffer_size = 256M
sort_buffer_size = 16M
#key_buffer = 128M
max_allowed_packet = 64M
thread_stack = 192K
!includedir /etc/my.cnf.d/
!includedir /etc/percona-xtradb-cluster.conf.d/
type or paste code here
type or paste code here