PXC IST Recovery Time After Heavy Write Load (11+ Hours of Continuous Writes)

MySQL & MariaDB Percona XtraDB Cluster 5.x
1

Hello Percona Team,

We are testing Percona XtraDB Cluster (PXC) 8.4.x with the objective of understanding the recovery time after a node has been offline during a heavy write workload.

Test Environment

  • Percona XtraDB Cluster: 3 nodes
  • PXC Version: 8.4.x
  • Kubernetes (RKE2)
  • Continuous workload generated with 1000TPS using Sysbench
  • Approximately 11 hours of continuous write activity

Test Scenario

  1. Started a continuous Sysbench write workload.
  2. Shut down one PXC node.
  3. Kept the workload running for approximately 11 hours.
  4. Brought the node back online.
  5. The node recovered using IST (Incremental State Transfer) instead of SST.
  6. The synchronization completed much faster than we expected.

Our Understanding

Our understanding is that:

  • PXC uses Galera replication.
  • IST uses the gcache to transfer only the missing write-sets.
  • It does not replay MySQL binary logs.
  • Therefore, even after a large number of writes, recovery can still be very fast as long as the required write-sets are still available in the donor’s gcache.

Current Cluster Information

Current wsrep status shows:

  • wsrep_local_state_comment = Synced

  • wsrep_cluster_size = 3

  • wsrep_cluster_status = Primary

  • wsrep_provider_version = 4.24

  • wsrep_gcache_pool_size = 327056696 (approximately 312 MB)

  • wsrep_local_cached_downto = 140433700

  • wsrep_last_committed = 140547929
    gcache size - 10GB

Our Questions

  1. Is our understanding correct?
  2. Is it expected that a node can recover very quickly after approximately 11 hours of continuous writes if IST is used?
  3. Does the recovery time depend primarily on the amount of missing write-set data rather than the duration for which the node was offline?
  4. Am I missing anything in my understanding of how IST recovery works?
  5. Are there any recommended methods for accurately testing recovery time after a very large write workload?

We would appreciate any clarification or best practices from the community.

Thank you!

2

Here are the logs-

2026-08-19T05:17:19.554264Z 0 [Note] [MY-000000] [Galera] wsrep_load(): loading provider library ‘/usr/lib64/galera4/libgalera_smm.so’
2026-08-19T05:17:19.554975Z 0 [Note] [MY-000000] [Galera] wsrep_load(): Galera 4.24(a430f07) by Codership Oy info@codership.com (modified by Percona https://percona.com/) loaded successfully.
2026-08-19T05:17:19.554993Z 0 [Note] [MY-000000] [Galera] Resolved symbol ‘wsrep_node_isolation_mode_set_v1’
2026-08-19T05:17:19.554998Z 0 [Note] [MY-000000] [Galera] Resolved symbol ‘wsrep_certify_v1’
2026-08-19T05:17:19.555002Z 0 [Note] [MY-000000] [Galera] Initializing config service v2
2026-08-19T05:17:19.555242Z 0 [Note] [MY-000000] [Galera] Deinitializing config service v2
2026-08-19T05:17:19.555268Z 0 [Note] [MY-000000] [Galera] CRC-32C: using 64-bit x86 acceleration.
2026-08-19T05:17:19.555398Z 0 [Note] [MY-000000] [Galera] not using SSL compression
2026-08-19T05:17:19.555895Z 0 [Note] [MY-000000] [Galera] Found saved state: 7cb3e8b0-7149-11f1-b8df-26bb63d99319:140547925, safe_to_bootstrap: 0
2026-08-19T05:17:19.556987Z 0 [Note] [MY-000000] [Galera] GCache DEBUG: opened preamble:
Version: 2
UUID: 7cb3e8b0-7149-11f1-b8df-26bb63d99319
Seqno: 140433700 - 140547925
Offset: 121950336
Synced: 1
EncVersion: 1
Encrypted: 0
MasterKeyConst UUID: 7cb05436-7149-11f1-91a3-72f3100dbcc8
MasterKey UUID: 00000000-0000-0000-0000-000000000000
MasterKey ID: 0
2026-08-19T05:17:19.557004Z 0 [Note] [MY-000000] [Galera] Recovering GCache ring buffer: version: 2, UUID: 7cb3e8b0-7149-11f1-b8df-26bb63d99319, offset: 121950336
2026-08-19T05:17:19.557053Z 0 [Note] [MY-000000] [Galera] GCache::RingBuffer initial scan… 0.0% (0/10737418264 bytes) complete.
2026-08-19T05:17:19.984982Z 0 [Note] [MY-000000] [Galera] GCache::RingBuffer initial scan… 100.0% (10737418264/10737418264 bytes) complete.
2026-08-19T05:17:19.986400Z 0 [Note] [MY-000000] [Galera] Recovering GCache ring buffer: found gapless sequence 140433700-140547925
2026-08-19T05:17:19.986450Z 0 [Note] [MY-000000] [Galera] GCache::RingBuffer unused buffers scan… 0.0% (0/205103168 bytes) complete.
2026-08-19T05:17:19.989060Z 0 [Note] [MY-000000] [Galera] GCache::RingBuffer unused buffers scan… 100.0% (205103168/205103168 bytes) complete.
2026-08-19T05:17:19.989075Z 0 [Note] [MY-000000] [Galera] Recovering GCache ring buffer: found 2/114228 locked buffers
2026-08-19T05:17:19.989079Z 0 [Note] [MY-000000] [Galera] Recovering GCache ring buffer: free space: 10532315504/10737418240
2026-08-19T05:17:19.990004Z 0 [Note] [MY-000000] [Galera] Passing config to GCS: allocator.disk_pages_encryption = no; allocator.encryption_cache_page_size = 32K; allocator.encryption_cache_size = 16777216; base_dir = /var/lib/mysql/; base_host = 10.42.209.202; base_port = 4567; cert.log_conflicts = no; cert.optimistic_pa = no; debug = no; evs.auto_evict = 0; evs.causal_keepalive_period = PT1S; evs.debug_log_mask = 0x1; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.info_log_mask = 0; evs.join_retrans_period = PT1S; evs.keepalive_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 10; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.use_aggregate = true; evs.user_send_window = 4; evs.version = 1; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.encryption = no; gcache.encryption_cache_page_size = 32K; gcache.encryption_cache_size = 16777216; gcache.freeze_purge_at_seqno = -1; gcache.keep_pages_count = 0; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = galera.cache; gcache.page_size = 128M; gcache.recover = yes; gcache.size = 10G; gcomm.thread_prio = ; gcs.check_appl_proto = 1; gcs.fc_auto_evict_threshold = 0.75; gcs.fc_auto_evict_window = 0; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 100; gcs.fc_master_slave = no; gcs.fc_single_primary = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.mcast_ttl = 1; gmcast.peer_timeout = PT3S; gmcast.segment = 0; gmcast.time_wait = PT5S; gmcast.version = 0; pc.announce_timeout = PT3S; pc.checksum = false; pc.ignore_quorum = false; pc.ignore_sb = false; pc.linger = PT20S; pc.npvo = false; pc.recovery = true; pc.version = 0; pc.wait_prim = true; pc.wait_prim_timeout = PT30S; pc.wait_restored_prim_timeout = PT0S; pc.weight = 1; protonet.backend = asio; protonet.version = 0; repl.causal_read_timeout = PT30S; repl.commit_order = 3; repl.key_format = FLAT8; repl.max_ws_size = 2147483647; repl.proto_max = 11; socket.checksum = 2; socket.recv_buf_size = auto; socket.send_buf_size = auto; socket.ssl = YES; socket.ssl_ca = /etc/mysql/ssl-internal/ca.crt; socket.ssl_cert = /etc/mysql/ssl-internal/tls.crt; socket.ssl_cipher = ; socket.ssl_key = /etc/mysql/ssl-internal/tls.key; socket.ssl_reload = 1;
2026-08-19T05:17:20.004419Z 0 [Note] [MY-000000] [Galera] Service thread queue flushed.
2026-08-19T05:17:20.004482Z 0 [Note] [MY-000000] [Galera] ####### Assign initial position for certification: 7cb3e8b0-7149-11f1-b8df-26bb63d99319:140547925, protocol version: -1
2026-08-19T05:17:20.004576Z 0 [Note] [MY-000000] [WSREP] Starting replication
2026-08-19T05:17:20.004592Z 0 [Note] [MY-000000] [Galera] Connecting with bootstrap option: 0
2026-08-19T05:17:20.004598Z 0 [Note] [MY-000000] [Galera] Setting GCS initial position t

3
  1. Is our understanding correct?

Yes, all 4 points are correct.

  1. Is it expected that a node can recover very quickly after approximately 11 hours of continuous writes if IST is used?

Yes. Even days of writes.

  1. Does the recovery time depend primarily on the amount of missing write-set data rather than the duration for which the node was offline?

It depends 100% on the amount of missing write-set data, and has absolutely nothing to do with time/duration.

  1. Am I missing anything in my understanding of how IST recovery works?

Nope. All good.

  1. Are there any recommended methods for accurately testing recovery time after a very large write workload?

Your method is pretty spot on.

Check out gcache.freeze_purge_at_seqno which gives the ability for gcache to grow indefinitely if needed in certain situations.

4

Thank you so much Sir!!