hi,all,when we use percona cluster in our pruduction box,we away got this msg in error log.what is it mean?
2017-08-20 04:02:30 29346 [Note] WSREP: Provider resumed.
2017-08-21 04:02:36 29346 [Note] WSREP: Provider paused at 71ff9701-81a2-11e7-aa25-5f865e134a58:313477 (287475)
2017-08-21 04:02:42 29346 [Note] WSREP: resuming provider at 287475
2017-08-21 04:02:42 29346 [Note] WSREP: Provider resumed.
2017-08-22 04:03:48 29346 [Note] WSREP: Provider paused at 71ff9701-81a2-11e7-aa25-5f865e134a58:389024 (364778)
2017-08-22 04:03:54 29346 [Note] WSREP: resuming provider at 364778
2017-08-22 04:03:54 29346 [Note] WSREP: Provider resumed.
2017-08-23 04:04:15 29346 [Note] WSREP: Provider paused at 71ff9701-81a2-11e7-aa25-5f865e134a58:461257 (438699)
2017-08-23 04:04:20 29346 [Note] WSREP: resuming provider at 438699
2017-08-23 04:04:20 29346 [Note] WSREP: Provider resumed.
Hello,
That could be caused by flow control. Please read the following link:
[url]http://galeracluster.com/documentation-webpages/nodestates.html[/url]
[url]http://galeracluster.com/documentation-webpages/managingfc.html[/url]
So, a node could have a large queue of transactions to apply and therefore is asking the other nodes to pause.
------------------------------±-------------------------------------+
| wsrep_local_state_uuid | 71ff9701-81a2-11e7-aa25-5f865e134a58 |
| wsrep_protocol_version | 5 |
| wsrep_last_committed | 602801 |
| wsrep_replicated | 302309 |
| wsrep_replicated_bytes | 7776478820 |
| wsrep_repl_keys | 17804807 |
| wsrep_repl_keys_bytes | 149561272 |
| wsrep_repl_data_bytes | 7101450919 |
| wsrep_repl_other_bytes | 0 |
| wsrep_received | 281437 |
| wsrep_received_bytes | 2788159496 |
| wsrep_local_commits | 302288 |
| wsrep_local_cert_failures | 15 |
| wsrep_local_replays | 0 |
| wsrep_local_send_queue | 0 |
| wsrep_local_send_queue_avg | 0.000122 |
| wsrep_local_recv_queue | 0 |
| wsrep_local_recv_queue_avg | 62.377498 |
| wsrep_local_cached_downto | 476116 |
| wsrep_flow_control_paused_ns | 626083319 |
| wsrep_flow_control_paused | 0.000001 |
| wsrep_flow_control_sent | 0 |
| wsrep_flow_control_recv | 5 |
| wsrep_cert_deps_distance | 48.080979 |
| wsrep_apply_oooe | 0.012610 |
| wsrep_apply_oool | 0.000310 |
| wsrep_apply_window | 1.073725 |
| wsrep_commit_oooe | 0.000000 |
| wsrep_commit_oool | 0.000062 |
| wsrep_commit_window | 1.056073 |
| wsrep_local_state | 4 |
| wsrep_local_state_comment | Synced |
| wsrep_cert_index_size | 733 |
| wsrep_causal_reads | 64131943 |
| wsrep_cert_interval | 0.004100 |
| wsrep_incoming_addresses | , |
| wsrep_cluster_conf_id | 9 |
| wsrep_cluster_size | 3 |
| wsrep_cluster_state_uuid | 71ff9701-81a2-11e7-aa25-5f865e134a58 |
| wsrep_cluster_status | Primary |
| wsrep_connected | ON |
| wsrep_local_bf_aborts | 307 |
| wsrep_local_index | 1 |
| wsrep_provider_name | Galera |
| wsrep_provider_vendor | Codership Oy <info@codership.com> |
| wsrep_provider_version | 3.5(r178) |
| wsrep_ready | ON
i have post our cluster status pamerater,can you have a look it,is my cluster going to bad ???
From the status I see that flow control happens in your cluster:
| wsrep_local_recv_queue_avg | 62.377498 |
| wsrep_local_cached_downto | 476116 |
| wsrep_flow_control_paused_ns | 626083319 |
| wsrep_flow_control_paused | 0.000001 |
| wsrep_flow_control_sent | 0 |
| wsrep_flow_control_recv | 5 |
and also, that since you are writing to multiple nodes at the same time there are many deadlocks errors:
| wsrep_local_bf_aborts | 307 |
So, I would recommend you to write on a single node and also increase gcs.fc_limit in case you have it configured to its default value. That variable defines the number of transactions in the queue that will trigger flow control. The default number is not usually good for production.
did flow control happens will cause cluster going to bad?how to optimize it?
When flow control happens the nodes pause and stop applying transactions, so the throughput is lower. To reduce it, you can configure the thresholds that trigger the flow control as explained in the documentation links I shared. If your servers are VM, you can also increase the resources. In a nutshell, flow control means “I cannot apply the changes as fast as I receive them”. Think that it is something similar to regular async replication. I recommend you again to read the previous two links, everything is explained there.