Hi , i have a 3 node Percona Mysql cluster 57-5.7.25-31 . Unfortunately i’ve lost one node and i had to rebuild competly by WSREP layer ( node after sst became synced with others )
Since the recover i get high values of wsrep_cert_deps_distance and wsrep_local_recv_queue_avg but decreasing .
Afaik these values has to decrease to 0.X values ( as like other nodes ) .
I would like to know if it safe wait reduction of these values before to enable the node to application .
Current status
node 1 : wsrep_cert_deps_distance | 588.899176
wsrep_local_recv_queue_avg | 59009.332682 - initial value after SST 362700.476338
node 2 : wsrep_cert_deps_distance | 52.880276
wsrep_local_recv_queue_avg | 0.031238
node 3 : wsrep_cert_deps_distance | 52.870661
wsrep_local_recv_queue_avg | 0.634849
All nodes are synced with none latency and spikes on flow control
There are two status variables you can use to find slow nodes in a cluster: wsrep_flow_control_sent and wsrep_local_recv_queue_avg. Check these status variables on each node in a cluster. The node that returns the highest value is the slowest one.
The wsrep_flow_control_sent variable provides the number of times a node sent a pause event due to flow control since the last status query. The wsrep_local_recv_queue_avg varaible returns an average of the received queue length since the last status query. Nodes that return values much higher than 0.0 indicate that it cannot apply write-sets as fast as they are received and can generate replication throttling.
Check these status variables on each node in your cluster. The node that returns the highest value is the slowest node. Lower values are preferable.
Hi @dclem thanks for posting to the Percona forums!
It appears that node 1 is unable to apply the volume of data changes in the cluster as indicated by high wsrep_local_recv_queue_avg . What makes this node unique in your deployment? Does it have less vCPU than the other nodes, or slower disk?
If you use PMM, from the PXC/Galera Cluster Summary dashboard, can share with us the graphs:
No ,nothing different than other nodes wsrep_local_recv_queue_avg as the node recovered from SST started with high values ,then decreasing and still decreasing ,now
±-------------------------±-----------+
| Variable_name | Value |
±-------------------------±-----------+
| wsrep_cert_deps_distance | 251.988942 |
±-------------------------±-----------+
1 row in set (0.00 sec)
