Not the answer you need?
Register and ask your own question!

what is "Provider paused at "mean

dixondixon ContributorInactive User Role Beginner
hi,all,when we use percona cluster in our pruduction box,we away got this msg in error log.what is it mean?
2017-08-20 04:02:30 29346 [Note] WSREP: Provider resumed.
2017-08-21 04:02:36 29346 [Note] WSREP: Provider paused at 71ff9701-81a2-11e7-aa25-5f865e134a58:313477 (287475)
2017-08-21 04:02:42 29346 [Note] WSREP: resuming provider at 287475
2017-08-21 04:02:42 29346 [Note] WSREP: Provider resumed.
2017-08-22 04:03:48 29346 [Note] WSREP: Provider paused at 71ff9701-81a2-11e7-aa25-5f865e134a58:389024 (364778)
2017-08-22 04:03:54 29346 [Note] WSREP: resuming provider at 364778
2017-08-22 04:03:54 29346 [Note] WSREP: Provider resumed.
2017-08-23 04:04:15 29346 [Note] WSREP: Provider paused at 71ff9701-81a2-11e7-aa25-5f865e134a58:461257 (438699)
2017-08-23 04:04:20 29346 [Note] WSREP: resuming provider at 438699
2017-08-23 04:04:20 29346 [Note] WSREP: Provider resumed.

Comments

  • miguelangelnietomiguelangelnieto Member Inactive User Role Beginner
    Hello,

    That could be caused by flow control. Please read the following link:

    http://galeracluster.com/documentation-webpages/nodestates.html
    http://galeracluster.com/documentation-webpages/managingfc.html

    So, a node could have a large queue of transactions to apply and therefore is asking the other nodes to pause.
  • dixondixon Contributor Inactive User Role Beginner
    Hello,

    That could be caused by flow control. Please read the following link:

    http://galeracluster.com/documentation-webpages/nodestates.html
    http://galeracluster.com/documentation-webpages/managingfc.html

    So, a node could have a large queue of transactions to apply and therefore is asking the other nodes to pause.

    +
    +
    | wsrep_local_state_uuid | 71ff9701-81a2-11e7-aa25-5f865e134a58 |
    | wsrep_protocol_version | 5 |
    | wsrep_last_committed | 602801 |
    | wsrep_replicated | 302309 |
    | wsrep_replicated_bytes | 7776478820 |
    | wsrep_repl_keys | 17804807 |
    | wsrep_repl_keys_bytes | 149561272 |
    | wsrep_repl_data_bytes | 7101450919 |
    | wsrep_repl_other_bytes | 0 |
    | wsrep_received | 281437 |
    | wsrep_received_bytes | 2788159496 |
    | wsrep_local_commits | 302288 |
    | wsrep_local_cert_failures | 15 |
    | wsrep_local_replays | 0 |
    | wsrep_local_send_queue | 0 |
    | wsrep_local_send_queue_avg | 0.000122 |
    | wsrep_local_recv_queue | 0 |
    | wsrep_local_recv_queue_avg | 62.377498 |
    | wsrep_local_cached_downto | 476116 |
    | wsrep_flow_control_paused_ns | 626083319 |
    | wsrep_flow_control_paused | 0.000001 |
    | wsrep_flow_control_sent | 0 |
    | wsrep_flow_control_recv | 5 |
    | wsrep_cert_deps_distance | 48.080979 |
    | wsrep_apply_oooe | 0.012610 |
    | wsrep_apply_oool | 0.000310 |
    | wsrep_apply_window | 1.073725 |
    | wsrep_commit_oooe | 0.000000 |
    | wsrep_commit_oool | 0.000062 |
    | wsrep_commit_window | 1.056073 |
    | wsrep_local_state | 4 |
    | wsrep_local_state_comment | Synced |
    | wsrep_cert_index_size | 733 |
    | wsrep_causal_reads | 64131943 |
    | wsrep_cert_interval | 0.004100 |
    | wsrep_incoming_addresses | ,, |
    | wsrep_cluster_conf_id | 9 |
    | wsrep_cluster_size | 3 |
    | wsrep_cluster_state_uuid | 71ff9701-81a2-11e7-aa25-5f865e134a58 |
    | wsrep_cluster_status | Primary |
    | wsrep_connected | ON |
    | wsrep_local_bf_aborts | 307 |
    | wsrep_local_index | 1 |
    | wsrep_provider_name | Galera |
    | wsrep_provider_vendor | Codership Oy <[email protected]> |
    | wsrep_provider_version | 3.5(r178) |
    | wsrep_ready | ON
  • dixondixon Contributor Inactive User Role Beginner
    Hello,

    That could be caused by flow control. Please read the following link:

    http://galeracluster.com/documentation-webpages/nodestates.html
    http://galeracluster.com/documentation-webpages/managingfc.html

    So, a node could have a large queue of transactions to apply and therefore is asking the other nodes to pause.

    i have post our cluster status pamerater,can you have a look it,is my cluster going to bad ???
  • miguelangelnietomiguelangelnieto Member Inactive User Role Beginner
    From the status I see that flow control happens in your cluster:

    | wsrep_local_recv_queue_avg | 62.377498 |
    | wsrep_local_cached_downto | 476116 |
    | wsrep_flow_control_paused_ns | 626083319 |
    | wsrep_flow_control_paused | 0.000001 |
    | wsrep_flow_control_sent | 0 |
    | wsrep_flow_control_recv | 5 |

    and also, that since you are writing to multiple nodes at the same time there are many deadlocks errors:

    | wsrep_local_bf_aborts | 307 |

    So, I would recommend you to write on a single node and also increase gcs.fc_limit in case you have it configured to its default value. That variable defines the number of transactions in the queue that will trigger flow control. The default number is not usually good for production.
  • dixondixon Contributor Inactive User Role Beginner
    From the status I see that flow control happens in your cluster:

    | wsrep_local_recv_queue_avg | 62.377498 |
    | wsrep_local_cached_downto | 476116 |
    | wsrep_flow_control_paused_ns | 626083319 |
    | wsrep_flow_control_paused | 0.000001 |
    | wsrep_flow_control_sent | 0 |
    | wsrep_flow_control_recv | 5 |

    and also, that since you are writing to multiple nodes at the same time there are many deadlocks errors:

    | wsrep_local_bf_aborts | 307 |

    So, I would recommend you to write on a single node and also increase gcs.fc_limit in case you have it configured to its default value. That variable defines the number of transactions in the queue that will trigger flow control. The default number is not usually good for production.
    did flow control happens will cause cluster going to bad?how to optimize it?
  • miguelangelnietomiguelangelnieto Member Inactive User Role Beginner
    When flow control happens the nodes pause and stop applying transactions, so the throughput is lower. To reduce it, you can configure the thresholds that trigger the flow control as explained in the documentation links I shared. If your servers are VM, you can also increase the resources. In a nutshell, flow control means "I cannot apply the changes as fast as I receive them". Think that it is something similar to regular async replication. I recommend you again to read the previous two links, everything is explained there.
Sign In or Register to comment.

MySQL, InnoDB, MariaDB and MongoDB are trademarks of their respective owners.
Copyright ©2005 - 2020 Percona LLC. All rights reserved.