Hi all,
My cluster gets corrupted after running a failed CREATE PROCEDURE statement.
I’m having a 3 nodes Galera Cluster. On top of those 3 nodes, I have a proxysql:
mysql> select * from runtime_mysql_servers;
+--------------+----------+------+-----------+---------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| hostgroup_id | hostname | port | gtid_port | status | weight | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment |
+--------------+----------+------+-----------+---------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| 10 | prod-db-01 | 3306 | 0 | SHUNNED | 10000 | 0 | 1000 | 0 | 0 | 0 | |
| 10 | prod-db-02 | 3306 | 0 | SHUNNED | 10000 | 0 | 1000 | 0 | 0 | 0 | |
| 10 | prod-db-03 | 3306 | 0 | ONLINE | 10000 | 0 | 1000 | 0 | 0 | 0 | |
| 20 | prod-db-01 | 3306 | 0 | ONLINE | 10000 | 0 | 1000 | 0 | 0 | 0 | |
| 20 | prod-db-02 | 3306 | 0 | ONLINE | 10000 | 0 | 1000 | 0 | 0 | 0 | |
| 30 | prod-db-01 | 3306 | 0 | ONLINE | 10000 | 0 | 1000 | 0 | 0 | 0 | |
| 30 | prod-db-02 | 3306 | 0 | ONLINE | 10000 | 0 | 1000 | 0 | 0 | 0 | |
| 30 | prod-db-03 | 3306 | 0 | ONLINE | 10000 | 0 | 1000 | 0 | 0 | 0 | |
+--------------+----------+------+-----------+---------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
8 rows in set (0.00 sec)
I applied write/read split:
mysql> select * from runtime_mysql_query_rules\G;
*************************** 1. row ***************************
rule_id: 100
active: 1
username: NULL
schemaname: NULL
flagIN: 0
client_addr: NULL
proxy_addr: NULL
proxy_port: NULL
digest: NULL
match_digest: NULL
match_pattern: ^SELECT .* FOR UPDATE
negate_match_pattern: 0
re_modifiers: CASELESS
flagOUT: NULL
replace_pattern: NULL
destination_hostgroup: 10
cache_ttl: NULL
cache_empty_result: NULL
cache_timeout: NULL
reconnect: NULL
timeout: NULL
retries: NULL
delay: NULL
next_query_flagIN: NULL
mirror_flagOUT: NULL
mirror_hostgroup: NULL
error_msg: NULL
OK_msg: NULL
sticky_conn: NULL
multiplex: NULL
gtid_from_hostgroup: NULL
log: NULL
apply: 1
attributes:
comment: NULL
*************************** 2. row ***************************
rule_id: 200
active: 1
username: NULL
schemaname: NULL
flagIN: 0
client_addr: NULL
proxy_addr: NULL
proxy_port: NULL
digest: NULL
match_digest: NULL
match_pattern: ^SELECT .*
negate_match_pattern: 0
re_modifiers: CASELESS
flagOUT: NULL
replace_pattern: NULL
destination_hostgroup: 30
cache_ttl: NULL
cache_empty_result: NULL
cache_timeout: NULL
reconnect: NULL
timeout: NULL
retries: NULL
delay: NULL
next_query_flagIN: NULL
mirror_flagOUT: NULL
mirror_hostgroup: NULL
error_msg: NULL
OK_msg: NULL
sticky_conn: NULL
multiplex: NULL
gtid_from_hostgroup: NULL
log: NULL
apply: 1
attributes:
comment: NULL
*************************** 3. row ***************************
rule_id: 300
active: 1
username: NULL
schemaname: NULL
flagIN: 0
client_addr: NULL
proxy_addr: NULL
proxy_port: NULL
digest: NULL
match_digest: NULL
match_pattern: .*
negate_match_pattern: 0
re_modifiers: CASELESS
flagOUT: NULL
replace_pattern: NULL
destination_hostgroup: 10
cache_ttl: NULL
cache_empty_result: NULL
cache_timeout: NULL
reconnect: NULL
timeout: NULL
retries: NULL
delay: NULL
next_query_flagIN: NULL
mirror_flagOUT: NULL
mirror_hostgroup: NULL
error_msg: NULL
OK_msg: NULL
sticky_conn: NULL
multiplex: NULL
gtid_from_hostgroup: NULL
log: NULL
apply: 1
attributes:
comment: NULL
3 rows in set (0.00 sec)
ERROR:
No query specified
The problem is that after running a failed CREATE PROCEDURE satement (it goes to ProxySQL first then gets routed to Database node), my cluster gets corrupted.
This is the log from Node01:
PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
final: yes
own_index: -1
members(0):
=================================================
2024-02-08T07:40:12.820619Z 10 [Note] [MY-000000] [Galera] Non-primary view
2024-02-08T07:40:12.820631Z 10 [Note] [MY-000000] [WSREP] Server status change connected -> disconnected
2024-02-08T07:40:12.820641Z 10 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2024-02-08T07:40:12.820653Z 10 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2024-02-08T07:40:12.820674Z 10 [Note] [MY-000000] [Galera] Waiting 600 seconds for 4 receivers to finish
2024-02-08T07:40:12.829468Z 12 [Note] [MY-000000] [Galera] Slave thread exit. Return code: 6
2024-02-08T07:40:12.829549Z 12 [Note] [MY-000000] [WSREP] Applier thread exiting ret: 6 thd: 12
2024-02-08T07:40:12.829723Z 2 [Note] [MY-000000] [Galera] Slave thread exit. Return code: 6
2024-02-08T07:40:12.829751Z 2 [Note] [MY-000000] [WSREP] Applier thread exiting ret: 6 thd: 2
2024-02-08T07:40:12.829750Z 11 [Note] [MY-000000] [Galera] Slave thread exit. Return code: 6
2024-02-08T07:40:12.829795Z 11 [Note] [MY-000000] [WSREP] Applier thread exiting ret: 6 thd: 11
2024-02-08T07:40:12.833880Z 0 [Note] [MY-000000] [Galera] Service thread queue flushed.
2024-02-08T07:40:12.834008Z 10 [Note] [MY-000000] [Galera] ####### Assign initial position for certification: 00000000-0000-0000-0000-000000000000:-1, protocol version: 5
2024-02-08T07:40:12.834039Z 10 [Note] [MY-000000] [Galera] Slave thread exit. Return code: 0
2024-02-08T07:40:12.834054Z 10 [Note] [MY-000000] [WSREP] Applier thread exiting ret: 0 thd: 10
This is the log from Node02:
2024-02-08T07:34:15.274695Z 11 [ERROR] [MY-010584] [Repl] Replica SQL: Error 'PROCEDURE GetRowCounts already exists' on query. Default database: 'lims-staging'. Query: 'CREATE DEFINER=`myuser`@`%` PROCEDURE `GetRowCounts`() BEGIN DECLARE done INT DEFAULT FALSE; DECLARE tableName VARCHAR(255); DECLARE cur CURSOR FOR SELECT table_name FROM information_schema.tables WHERE table_schema = 'your_database_name'; DECLARE CONTINUE HANDLER FOR NOT FOUND SET done = TRUE; -- Create a temporary table to store the results CREATE TEMPORARY TABLE IF NOT EXISTS temp_row_counts (Table_Name VARCHAR(255), Row_Count INT); OPEN cur; read_loop: LOOP FETCH cur INTO tableName; IF done THEN LEAVE read_loop; END IF; SET @query = CONCAT('SELECT "', tableName, '" AS Table_Name, COUNT(*) AS Row_Count FROM ', tableName); PREPARE stmt FROM @query; EXECUTE stmt; -- Insert the result into the temporary table INSERT INTO temp_row_counts (Table_Name, Row_Count) VALUES (tableName, @Row_Count); DE, Error_code: MY-001304
2024-02-08T07:34:15.274830Z 11 [Warning] [MY-000000] [WSREP] Event 1 Query apply failed: 1, seqno 79454588
2024-02-08T07:34:15.278097Z 0 [Note] [MY-000000] [Galera] Member 1(192.168.101.12) initiates vote on df948d8d-5994-11ec-91b5-ff5ae1e8c4ba:79454588,c0a03718190410cb: PROCEDURE GetRowCounts already exists, Error_code: 1304;
2024-02-08T07:34:15.278347Z 0 [Note] [MY-000000] [Galera] Recomputed vote based on error codes: 1304. New vote c435e70801d6f1d3 will be used for further steps. Old Vote: c0a03718190410cb
2024-02-08T07:34:15.278400Z 0 [Note] [MY-000000] [Galera] Votes over df948d8d-5994-11ec-91b5-ff5ae1e8c4ba:79454588:
c435e70801d6f1d3: 1/3
Waiting for more votes.
2024-02-08T07:34:15.278432Z 0 [Note] [MY-000000] [Galera] Member 2(192.168.101.13) initiates vote on df948d8d-5994-11ec-91b5-ff5ae1e8c4ba:79454588,cf8044408b48d83e: PROCEDURE lims-staging.CountNumberOfRowsInEachTable does not exist, Error_code: 1305; PROCEDURE GetRowCounts already exists, Error_code: 1304;
2024-02-08T07:34:15.278533Z 0 [Note] [MY-000000] [Galera] Recomputed vote based on error codes: 1304, 1305. New vote b9a165a15b5f2ced will be used for further steps. Old Vote: cf8044408b48d83e
2024-02-08T07:34:15.278561Z 0 [Note] [MY-000000] [Galera] Votes over df948d8d-5994-11ec-91b5-ff5ae1e8c4ba:79454588:
b9a165a15b5f2ced: 1/3
c435e70801d6f1d3: 1/3
Waiting for more votes.
2024-02-08T07:34:15.278708Z 0 [Note] [MY-000000] [Galera] Member 0(192.168.101.11) initiates vote on df948d8d-5994-11ec-91b5-ff5ae1e8c4ba:79454588,c0a03718190410cb: PROCEDURE GetRowCounts already exists, Error_code: 1304;
2024-02-08T07:34:15.278808Z 0 [Note] [MY-000000] [Galera] Recomputed vote based on error codes: 1304. New vote c435e70801d6f1d3 will be used for further steps. Old Vote: c0a03718190410cb
2024-02-08T07:34:15.278828Z 0 [Note] [MY-000000] [Galera] Votes over df948d8d-5994-11ec-91b5-ff5ae1e8c4ba:79454588:
b9a165a15b5f2ced: 1/3
c435e70801d6f1d3: 2/3
Winner: c435e70801d6f1d3
2024-02-08T07:34:15.282556Z 0 [Note] [MY-000000] [Galera] declaring 8326ec1d-9734 at ssl://192.168.101.11:4567 stable
2024-02-08T07:34:15.282722Z 0 [Note] [MY-000000] [Galera] forgetting d5569871-874b (ssl://192.168.101.13:4567)
2024-02-08T07:34:15.283143Z 0 [Note] [MY-000000] [Galera] Node 8326ec1d-9734 state primary
2024-02-08T07:34:15.285253Z 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node
view (view_id(PRIM,8326ec1d-9734,22)
memb {
8326ec1d-9734,0
b29296da-a514,0
}
joined {
}
left {
}
partitioned {
d5569871-874b,0
}
)
2024-02-08T07:34:15.285302Z 0 [Note] [MY-000000] [Galera] Save the discovered primary-component to disk
2024-02-08T07:34:15.286938Z 0 [Note] [MY-000000] [Galera] forgetting d5569871-874b (ssl://192.168.101.13:4567)
2024-02-08T07:34:15.286978Z 0 [Note] [MY-000000] [Galera] New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 2
2024-02-08T07:34:15.287020Z 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: Waiting for state UUID.
2024-02-08T07:34:15.287299Z 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: sent state msg: 75efa99b-c654-11ee-ba59-12a0efe68892
2024-02-08T07:34:15.287531Z 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: got state msg: 75efa99b-c654-11ee-ba59-12a0efe68892 from 0 (192.168.101.11)
2024-02-08T07:34:15.287574Z 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: got state msg: 75efa99b-c654-11ee-ba59-12a0efe68892 from 1 (192.168.101.12)
2024-02-08T07:34:15.287593Z 0 [Note] [MY-000000] [Galera] Quorum results:
version = 6,
component = PRIMARY,
conf_id = 21,
members = 2/2 (primary/total),
act_id = 79454588,
last_appl. = 79454561,
protocols = 2/10/4 (gcs/repl/appl),
vote policy= 0,
group UUID = df948d8d-5994-11ec-91b5-ff5ae1e8c4ba
2024-02-08T07:34:15.287662Z 0 [Note] [MY-000000] [Galera] Flow-control interval: [141, 141]
2024-02-08T07:34:15.287831Z 10 [Note] [MY-000000] [Galera] ####### processing CC 79454589, local, ordered
2024-02-08T07:34:15.287871Z 10 [Note] [MY-000000] [Galera] Maybe drain monitors from 79454588 upto current CC event 79454589 upto:79454588
2024-02-08T07:34:15.287890Z 10 [Note] [MY-000000] [Galera] Drain monitors from 79454588 up to 79454588
2024-02-08T07:34:15.287905Z 10 [Note] [MY-000000] [Galera] ####### My UUID: b29296da-c20b-11ee-a514-9b23a9ad452c
2024-02-08T07:34:15.287918Z 10 [Note] [MY-000000] [Galera] Skipping cert index reset
2024-02-08T07:34:15.287929Z 10 [Note] [MY-000000] [Galera] REPL Protocols: 10 (5)
2024-02-08T07:34:15.287942Z 10 [Note] [MY-000000] [Galera] ####### Adjusting cert position: 79454588 -> 79454589
2024-02-08T07:34:15.287996Z 0 [Note] [MY-000000] [Galera] Service thread queue flushed.
2024-02-08T07:34:15.289783Z 10 [Note] [MY-000000] [Galera] ================================================
View:
id: df948d8d-5994-11ec-91b5-ff5ae1e8c4ba:79454589
status: primary
protocol_version: 4
capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
final: no
own_index: 1
members(2):
0: 8326ec1d-c20b-11ee-9734-dedffa046b30, 192.168.101.11
1: b29296da-c20b-11ee-a514-9b23a9ad452c, 192.168.101.12
=================================================
2024-02-08T07:34:15.289866Z 10 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2024-02-08T07:34:15.294648Z 10 [Note] [MY-000000] [Galera] Recording CC from group: 79454589
2024-02-08T07:34:15.294681Z 10 [Note] [MY-000000] [Galera] Lowest cert index boundary for CC from group: 79454562
2024-02-08T07:34:15.294697Z 10 [Note] [MY-000000] [Galera] Min available from gcache for CC from group: 78768641
2024-02-08T07:34:20.586475Z 0 [Note] [MY-000000] [Galera] cleaning up d5569871-874b (ssl://192.168.101.13:4567)
2024-02-08T07:40:11.767605Z 0 [Note] [MY-000000] [Galera] Member 1(192.168.101.12) initiates vote on df948d8d-5994-11ec-91b5-ff5ae1e8c4ba:79455880,a41bd82a387e7d50: PROCEDURE lims-staging.GetRowCounters3 does not exist, Error_code: 1305; PROCEDURE GetRowCounts already exists, Error_code: 1304;
2024-02-08T07:40:11.767844Z 0 [Note] [MY-000000] [Galera] Recomputed vote based on error codes: 1304, 1305. New vote c30ab8e29f19e654 will be used for further steps. Old Vote: a41bd82a387e7d50
2024-02-08T07:40:11.767883Z 0 [Note] [MY-000000] [Galera] Votes over df948d8d-5994-11ec-91b5-ff5ae1e8c4ba:79455880:
c30ab8e29f19e654: 1/2
Waiting for more votes.
2024-02-08T07:40:11.768775Z 0 [Note] [MY-000000] [Galera] Member 0(192.168.101.11) initiates vote on df948d8d-5994-11ec-91b5-ff5ae1e8c4ba:79455880,a5e736dc0902a842: PROCEDURE GetRowCounts already exists, Error_code: 1304;
2024-02-08T07:40:11.768876Z 0 [Note] [MY-000000] [Galera] Recomputed vote based on error codes: 1304. New vote b828a5a84caece8b will be used for further steps. Old Vote: a5e736dc0902a842
2024-02-08T07:40:11.768896Z 0 [Note] [MY-000000] [Galera] Votes over df948d8d-5994-11ec-91b5-ff5ae1e8c4ba:79455880:
b828a5a84caece8b: 1/2
c30ab8e29f19e654: 1/2
Winner: c30ab8e29f19e654
2024-02-08T07:40:12.771056Z 0 [Note] [MY-000000] [Galera] forgetting 8326ec1d-9734 (ssl://192.168.101.11:4567)
2024-02-08T07:40:12.771138Z 0 [Note] [MY-000000] [Galera] forgetting 8326ec1d-9734 (ssl://192.168.101.151:4567)
2024-02-08T07:40:12.771224Z 0 [Note] [MY-000000] [Galera] Node b29296da-a514 state primary
2024-02-08T07:40:12.771338Z 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node
view (view_id(PRIM,b29296da-a514,23)
memb {
b29296da-a514,0
}
joined {
}
left {
}
partitioned {
8326ec1d-9734,0
}
)
2024-02-08T07:40:12.771410Z 0 [Note] [MY-000000] [Galera] Save the discovered primary-component to disk
2024-02-08T07:40:12.773526Z 0 [Note] [MY-000000] [Galera] forgetting 8326ec1d-9734 (ssl://192.168.101.11:4567)
2024-02-08T07:40:12.773550Z 0 [Note] [MY-000000] [Galera] forgetting 8326ec1d-9734 (ssl://192.168.101.151:4567)
2024-02-08T07:40:12.773663Z 0 [Note] [MY-000000] [Galera] New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 1
2024-02-08T07:40:12.773845Z 0 [Note] [MY-000000] [Galera] STATE_EXCHANGE: sent state UUID: 4afde6bb-c655-11ee-a73a-c6c4fecea641
2024-02-08T07:40:12.773964Z 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: sent state msg: 4afde6bb-c655-11ee-a73a-c6c4fecea641
2024-02-08T07:40:12.773998Z 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: got state msg: 4afde6bb-c655-11ee-a73a-c6c4fecea641 from 0 (192.168.101.12)
2024-02-08T07:40:12.774018Z 0 [Note] [MY-000000] [Galera] Quorum results:
version = 6,
component = PRIMARY,
conf_id = 22,
members = 1/1 (primary/total),
act_id = 79455880,
last_appl. = 79454561,
protocols = 2/10/4 (gcs/repl/appl),
vote policy= 0,
group UUID = df948d8d-5994-11ec-91b5-ff5ae1e8c4ba
2024-02-08T07:40:12.774075Z 0 [Note] [MY-000000] [Galera] Flow-control interval: [100, 100]
2024-02-08T07:40:12.774189Z 10 [Note] [MY-000000] [Galera] ####### processing CC 79455881, local, ordered
2024-02-08T07:40:12.774243Z 10 [Note] [MY-000000] [Galera] Maybe drain monitors from 79455880 upto current CC event 79455881 upto:79455880
2024-02-08T07:40:12.774262Z 10 [Note] [MY-000000] [Galera] Drain monitors from 79455880 up to 79455880
2024-02-08T07:40:12.774277Z 10 [Note] [MY-000000] [Galera] ####### My UUID: b29296da-c20b-11ee-a514-9b23a9ad452c
2024-02-08T07:40:12.774298Z 10 [Note] [MY-000000] [Galera] Skipping cert index reset
2024-02-08T07:40:12.774311Z 10 [Note] [MY-000000] [Galera] REPL Protocols: 10 (5)
2024-02-08T07:40:12.774327Z 10 [Note] [MY-000000] [Galera] ####### Adjusting cert position: 79455880 -> 79455881
2024-02-08T07:40:12.774384Z 0 [Note] [MY-000000] [Galera] Service thread queue flushed.
2024-02-08T07:40:12.776912Z 10 [Note] [MY-000000] [Galera] ================================================
View:
id: df948d8d-5994-11ec-91b5-ff5ae1e8c4ba:79455881
status: primary
protocol_version: 4
capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
final: no
own_index: 0
members(1):
0: b29296da-c20b-11ee-a514-9b23a9ad452c, 192.168.101.12
=================================================
2024-02-08T07:40:12.777026Z 10 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2024-02-08T07:40:12.780338Z 10 [Note] [MY-000000] [Galera] Recording CC from group: 79455881
2024-02-08T07:40:12.780375Z 10 [Note] [MY-000000] [Galera] Lowest cert index boundary for CC from group: 79454562
2024-02-08T07:40:12.780391Z 10 [Note] [MY-000000] [Galera] Min available from gcache for CC from group: 78768641
2024-02-08T07:40:18.131127Z 0 [Note] [MY-000000] [Galera] cleaning up 8326ec1d-9734 (ssl://192.168.101.11:4567)
2024-02-08T07:40:18.131231Z 0 [Note] [MY-000000] [Galera] cleaning up 8326ec1d-9734 (ssl://192.168.101.151:4567)
2024-02-08T07:47:49.125804Z 0 [Note] [MY-000000] [Galera] Member 0(192.168.101.12) initiates vote on df948d8d-5994-11ec-91b5-ff5ae1e8c4ba:79457941,e47904087886190f: PROCEDURE GetRowCounts already exists, Error_code: 1304;
2024-02-08T07:47:49.126053Z 0 [Note] [MY-000000] [Galera] Recomputed vote based on error codes: 1304. New vote dd1c31a3302c070f will be used for further steps. Old Vote: e47904087886190f
2024-02-08T07:47:49.126090Z 0 [Note] [MY-000000] [Galera] Votes over df948d8d-5994-11ec-91b5-ff5ae1e8c4ba:79457941:
dd1c31a3302c070f: 1/1
Winner: dd1c31a3302c070f
This is the log from Node03:
2024-02-08T07:34:15.245278Z 0 [Note] [MY-000000] [Galera] Member 1(192.168.101.12) initiates vote on df948d8d-5994-11ec-91b5-ff5ae1e8c4ba:79454588,c0a03718190410cb: PROCEDURE GetRowCounts already exists, Error_code: 1304;
2024-02-08T07:34:15.245489Z 0 [Note] [MY-000000] [Galera] Recomputed vote based on error codes: 1304. New vote c435e70801d6f1d3 will be used for further steps. Old Vote: c0a03718190410cb
2024-02-08T07:34:15.245538Z 0 [Note] [MY-000000] [Galera] Votes over df948d8d-5994-11ec-91b5-ff5ae1e8c4ba:79454588:
c435e70801d6f1d3: 1/3
Waiting for more votes.
2024-02-08T07:34:15.245585Z 0 [Note] [MY-000000] [Galera] Member 2(192.168.101.13) initiates vote on df948d8d-5994-11ec-91b5-ff5ae1e8c4ba:79454588,cf8044408b48d83e: PROCEDURE lims-staging.CountNumberOfRowsInEachTable does not exist, Error_code: 1305; PROCEDURE GetRowCounts already exists, Error_code: 1304;
2024-02-08T07:34:15.245619Z 19 [Note] [MY-000000] [Galera] Got vote request for seqno df948d8d-5994-11ec-91b5-ff5ae1e8c4ba:79454588
2024-02-08T07:34:15.245692Z 0 [Note] [MY-000000] [Galera] Recomputed vote based on error codes: 1304, 1305. New vote b9a165a15b5f2ced will be used for further steps. Old Vote: cf8044408b48d83e
2024-02-08T07:34:15.245714Z 0 [Note] [MY-000000] [Galera] Votes over df948d8d-5994-11ec-91b5-ff5ae1e8c4ba:79454588:
b9a165a15b5f2ced: 1/3
c435e70801d6f1d3: 1/3
Waiting for more votes.
2024-02-08T07:34:15.245799Z 0 [Note] [MY-000000] [Galera] Member 0(192.168.101.11) initiates vote on df948d8d-5994-11ec-91b5-ff5ae1e8c4ba:79454588,c0a03718190410cb: PROCEDURE GetRowCounts already exists, Error_code: 1304;
2024-02-08T07:34:15.245901Z 0 [Note] [MY-000000] [Galera] Recomputed vote based on error codes: 1304. New vote c435e70801d6f1d3 will be used for further steps. Old Vote: c0a03718190410cb
2024-02-08T07:34:15.245923Z 0 [Note] [MY-000000] [Galera] Votes over df948d8d-5994-11ec-91b5-ff5ae1e8c4ba:79454588:
b9a165a15b5f2ced: 1/3
c435e70801d6f1d3: 2/3
Winner: c435e70801d6f1d3
2024-02-08T07:34:15.246023Z 190635 [ERROR] [MY-000000] [Galera] Inconsistency detected: Inconsistent by consensus on df948d8d-5994-11ec-91b5-ff5ae1e8c4ba:79454588
at /mnt/jenkins/workspace/pxc80-autobuild-RELEASE/test/percona-xtradb-cluster-8.0.35-27/percona-xtradb-cluster-galera/galera/src/replicator_smm.cpp:process_apply_error():1454
2024-02-08T07:34:15.247525Z 190635 [Note] [MY-000000] [Galera] Closing send monitor...
2024-02-08T07:34:15.247564Z 190635 [Note] [MY-000000] [Galera] Closed send monitor.
2024-02-08T07:34:15.247586Z 190635 [Note] [MY-000000] [Galera] gcomm: terminating thread
2024-02-08T07:34:15.247611Z 190635 [Note] [MY-000000] [Galera] gcomm: joining thread
2024-02-08T07:34:15.248593Z 190635 [Note] [MY-000000] [Galera] gcomm: closing backend
2024-02-08T07:34:15.249359Z 190635 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node
view (view_id(NON_PRIM,8326ec1d-9734,21)
memb {
d5569871-874b,0
}
joined {
}
left {
}
partitioned {
8326ec1d-9734,0
b29296da-a514,0
}
)
2024-02-08T07:34:15.249455Z 190635 [Note] [MY-000000] [Galera] PC protocol downgrade 1 -> 0
2024-02-08T07:34:15.249498Z 190635 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node
view ((empty))
2024-02-08T07:34:15.249910Z 190635 [Note] [MY-000000] [Galera] gcomm: closed
2024-02-08T07:34:15.249965Z 0 [Note] [MY-000000] [Galera] New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
2024-02-08T07:34:15.250052Z 0 [Note] [MY-000000] [Galera] Flow-control interval: [100, 100]
2024-02-08T07:34:15.250070Z 0 [Note] [MY-000000] [Galera] Received NON-PRIMARY.
2024-02-08T07:34:15.250082Z 0 [Note] [MY-000000] [Galera] Shifting SYNCED -> OPEN (TO: 79454588)
2024-02-08T07:34:15.250108Z 0 [Note] [MY-000000] [Galera] New SELF-LEAVE.
2024-02-08T07:34:15.250183Z 0 [Note] [MY-000000] [Galera] Flow-control interval: [0, 0]
2024-02-08T07:34:15.250207Z 0 [Note] [MY-000000] [Galera] Received SELF-LEAVE. Closing connection.
2024-02-08T07:34:15.250221Z 0 [Note] [MY-000000] [Galera] Shifting OPEN -> CLOSED (TO: 79454588)
2024-02-08T07:34:15.250246Z 0 [Note] [MY-000000] [Galera] RECV thread exiting 0: Success
2024-02-08T07:34:15.250485Z 190635 [Note] [MY-000000] [Galera] recv_thread() joined.
2024-02-08T07:34:15.250506Z 190635 [Note] [MY-000000] [Galera] Closing replication queue.
2024-02-08T07:34:15.250518Z 190635 [Note] [MY-000000] [Galera] Closing slave action queue.
2024-02-08T07:34:15.250736Z 17 [Note] [MY-000000] [Galera] ================================================
View:
id: df948d8d-5994-11ec-91b5-ff5ae1e8c4ba:79454588
status: non-primary
protocol_version: 4
capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
final: no
own_index: 0
members(1):
0: d5569871-c20b-11ee-874b-de4394fead48, 192.168.101.13
=================================================
2024-02-08T07:34:15.250928Z 17 [Note] [MY-000000] [Galera] Non-primary view
2024-02-08T07:34:15.251000Z 17 [Note] [MY-000000] [WSREP] Server status change synced -> connected
2024-02-08T07:34:15.251117Z 17 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2024-02-08T07:34:15.251163Z 17 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2024-02-08T07:34:15.251243Z 17 [Note] [MY-000000] [Galera] ================================================
View:
id: df948d8d-5994-11ec-91b5-ff5ae1e8c4ba:79454588
status: non-primary
protocol_version: 4
capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
final: yes
own_index: -1
members(0):
=================================================
2024-02-08T07:34:15.251280Z 17 [Note] [MY-000000] [Galera] Non-primary view
2024-02-08T07:34:15.251293Z 17 [Note] [MY-000000] [WSREP] Server status change connected -> disconnected
2024-02-08T07:34:15.251303Z 17 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2024-02-08T07:34:15.251316Z 17 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2024-02-08T07:34:15.251341Z 17 [Note] [MY-000000] [Galera] Waiting 600 seconds for 4 receivers to finish
2024-02-08T07:34:15.260300Z 2 [Note] [MY-000000] [Galera] Slave thread exit. Return code: 6
2024-02-08T07:34:15.260314Z 18 [Note] [MY-000000] [Galera] Slave thread exit. Return code: 6
2024-02-08T07:34:15.260400Z 2 [Note] [MY-000000] [WSREP] Applier thread exiting ret: 6 thd: 2
2024-02-08T07:34:15.260434Z 18 [Note] [MY-000000] [WSREP] Applier thread exiting ret: 6 thd: 18
2024-02-08T07:34:15.260649Z 19 [Note] [MY-000000] [Galera] Slave thread exit. Return code: 6
2024-02-08T07:34:15.260676Z 19 [Note] [MY-000000] [WSREP] Applier thread exiting ret: 6 thd: 19
2024-02-08T07:34:15.261327Z 0 [Note] [MY-000000] [Galera] Service thread queue flushed.
2024-02-08T07:34:15.261419Z 17 [Note] [MY-000000] [Galera] ####### Assign initial position for certification: 00000000-0000-0000-0000-000000000000:-1, protocol version: 5
2024-02-08T07:34:15.261442Z 17 [Note] [MY-000000] [Galera] Slave thread exit. Return code: 0
2024-02-08T07:34:15.261457Z 17 [Note] [MY-000000] [WSREP] Applier thread exiting ret: 0 thd: 17
Thank you all.