We are facing an issue on a setup running percona operator v1.13.0 where the pxc cluster is unable to recover after being in a crashed state. The replica size was initially reduced from 3 to 2 and then back to 3. No logs available from the time crash happened to.
After reducing the size to 0 and then back to 3, current state is as below
$ kubectl get pods | grep percona
percona-db-pxc-db-haproxy-0 1/2 Running 0 4m33s
percona-db-pxc-db-pxc-0 2/3 Running 0 4m34s
pxc-db-pxc-0 logs in this state show below:
2025-02-21T04:57:32.611353Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2025-02-21T04:57:32.618032Z 0 [Note] mysqld (mysqld 5.7.36-39-57) starting as process 1 …
2025-02-21T04:57:32.628814Z 0 [Note] WSREP: Skipping automatic SSL certificate generation (enabled only in bootstrap mode)
2025-02-21T04:57:32.629224Z 0 [Note] WSREP: Setting wsrep_ready to false
2025-02-21T04:57:32.629241Z 0 [Note] WSREP: No pre-stored wsrep-start position found. Skipping position initialization.
2025-02-21T04:57:32.629247Z 0 [Note] WSREP: wsrep_load(): loading provider library ‘/usr/lib64/galera3/libgalera_smm.so’
2025-02-21T04:57:32.630172Z 0 [Note] WSREP: wsrep_load(): Galera 3.55(r8b6416d) by Codership Oy info@codership.com loaded successfully.
2025-02-21T04:57:32.630223Z 0 [Note] WSREP: CRC-32C: using 64-bit x86 acceleration.
2025-02-21T04:57:32.633481Z 0 [Note] WSREP: Found saved state: b0313e38-ece9-11ef-aa5a-06e8b3cecb90:0, safe_to_bootstrap: 1
2025-02-21T04:57:32.639459Z 0 [Note] WSREP: Passing config to GCS: base_dir = /var/lib/mysql/; base_host = 10.20.0.155; base_port = 4567; cert.log_conflicts = no; cert.optimistic_pa = yes; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 10; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 4; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.freeze_purge_at_seqno = -1; gcache.keep_pages_count = 0; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.recover = no; gcache.size = 128M; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1; gcs.fc_limit = 100; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc.checksum = false; pc.ignore_quorum = false; pc.ignore_sb = false; pc.npvo = false; pc.recovery = true; pc.version = 0; pc.wait_prim = true; pc.wait_prim_timeout = PT30S; pc.weight = 10; protonet.backend = asio; protonet.version = 0; repl.causal_read_timeout = PT30S; repl.commit_order = 3; repl.key_format = FLAT8; repl.max_ws_size = 2147483647; repl.proto_max = 9; socket.checksum = 2; socket.recv_buf_size = auto; socket.send_buf_size = auto; socket.ssl_ca = /etc/mysql/ssl-internal/ca.crt; socket.ssl_cert = /etc/mysql/ssl-internal/tls.crt; socket.ssl_cipher = ; socket.ssl_compression = YES; socket.ssl_key = /etc/mysql/ssl-internal/tls.key;
2025-02-21T04:57:32.655233Z 0 [Note] WSREP: Assign initial position for certification: 0, protocol version: -1
2025-02-21T04:57:32.655257Z 0 [Note] WSREP: Preparing to initiate SST/IST
2025-02-21T04:57:32.655261Z 0 [Note] WSREP: Starting replication
2025-02-21T04:57:32.655268Z 0 [Note] WSREP: Setting initial position to b0313e38-ece9-11ef-aa5a-06e8b3cecb90:0
2025-02-21T04:57:32.655601Z 0 [Note] WSREP: Using CRC-32C for message checksums.
2025-02-21T04:57:32.655646Z 0 [Note] WSREP: initializing ssl context
2025-02-21T04:57:32.655920Z 0 [Note] WSREP: gcomm thread scheduling priority set to other:0
2025-02-21T04:57:32.656001Z 0 [Warning] WSREP: Fail to access the file (/var/lib/mysql//gvwstate.dat) error (No such file or directory). It is possible if node is booting for first time or re-booting after a graceful shutdown
2025-02-21T04:57:32.656006Z 0 [Note] WSREP: Restoring primary-component from disk failed. Either node is booting for first time or re-booting after a graceful shutdown
2025-02-21T04:57:32.656148Z 0 [Note] WSREP: GMCast version 0
2025-02-21T04:57:32.656229Z 0 [Note] WSREP: (5c116011, ‘ssl://0.0.0.0:4567’) listening at ssl://0.0.0.0:4567
2025-02-21T04:57:32.656233Z 0 [Note] WSREP: (5c116011, ‘ssl://0.0.0.0:4567’) multicast: , ttl: 1
2025-02-21T04:57:32.656475Z 0 [Note] WSREP: EVS version 0
2025-02-21T04:57:32.656531Z 0 [Note] WSREP: gcomm: connecting to group ‘percona-db-pxc-db-pxc’, peer ‘’
2025-02-21T04:57:32.656553Z 0 [Note] WSREP: start_prim is enabled, turn off pc_recovery
2025-02-21T04:57:32.656806Z 0 [Note] WSREP: Node 5c116011 state primary
2025-02-21T04:57:32.656824Z 0 [Note] WSREP: Current view of cluster as seen by this node
view (view_id(PRIM,5c116011,1)
memb {
5c116011,0
}
joined {
}
left {
}
partitioned {
}
)
2025-02-21T04:57:32.656829Z 0 [Note] WSREP: Save the discovered primary-component to disk
2025-02-21T04:57:32.660368Z 0 [Note] WSREP: gcomm: connected
2025-02-21T04:57:32.660477Z 0 [Note] WSREP: Shifting CLOSED → OPEN (TO: 0)
2025-02-21T04:57:32.660634Z 0 [Note] WSREP: Waiting for SST/IST to complete.
2025-02-21T04:57:32.660914Z 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 1
2025-02-21T04:57:32.661187Z 0 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 5c122626-f010-11ef-b8be-166393090605
2025-02-21T04:57:32.661211Z 0 [Note] WSREP: STATE EXCHANGE: sent state msg: 5c122626-f010-11ef-b8be-166393090605
2025-02-21T04:57:32.661223Z 0 [Note] WSREP: STATE EXCHANGE: got state msg: 5c122626-f010-11ef-b8be-166393090605 from 0 (percona-db-pxc-db-pxc-0)
2025-02-21T04:57:32.661240Z 0 [Note] WSREP: Quorum results:
version = 6,
component = PRIMARY,
conf_id = 0,
members = 1/1 (primary/total),
act_id = 0,
last_appl. = -1,
protocols = 0/9/3 (gcs/repl/appl),
group UUID = b0313e38-ece9-11ef-aa5a-06e8b3cecb90
2025-02-21T04:57:32.661252Z 0 [Note] WSREP: Flow-control interval: [100, 100]
2025-02-21T04:57:32.661260Z 0 [Note] WSREP: Restored state OPEN → JOINED (0)
2025-02-21T04:57:32.661301Z 0 [Note] WSREP: Member 0.0 (percona-db-pxc-db-pxc-0) synced with group.
2025-02-21T04:57:32.661310Z 0 [Note] WSREP: Shifting JOINED → SYNCED (TO: 0)
2025-02-21T04:57:32.661511Z 2 [Note] WSREP: REPL Protocols: 9 (4, 2)
2025-02-21T04:57:32.661575Z 2 [Note] WSREP: REPL Protocols: 9 (4, 2)
2025-02-21T04:57:32.661600Z 2 [Note] WSREP: New cluster view: global state: b0313e38-ece9-11ef-aa5a-06e8b3cecb90:0, view# 1: Primary, number of nodes: 1, my index: 0, protocol version 3
2025-02-21T04:57:32.661608Z 2 [Note] WSREP: Setting wsrep_ready to true
2025-02-21T04:57:32.661715Z 0 [Note] WSREP: SST complete, seqno: 0
2025-02-21T04:57:32.663612Z 0 [Note] InnoDB: PUNCH HOLE support available
2025-02-21T04:57:32.663626Z 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2025-02-21T04:57:32.663629Z 0 [Note] InnoDB: Uses event mutexes
2025-02-21T04:57:32.663632Z 0 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2025-02-21T04:57:32.663635Z 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2025-02-21T04:57:32.663638Z 0 [Note] InnoDB: Using Linux native AIO
2025-02-21T04:57:32.664964Z 0 [Note] InnoDB: Number of pools: 1
2025-02-21T04:57:32.665118Z 0 [Note] InnoDB: Using CPU crc32 instructions
2025-02-21T04:57:32.670864Z 0 [Note] InnoDB: Initializing buffer pool, total size = 6.00781G, instances = 8, chunk size = 769M
2025-02-21T04:57:32.940254Z 0 [Note] InnoDB: Completed initialization of buffer pool
2025-02-21T04:57:33.035922Z 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2025-02-21T04:57:33.058964Z 0 [Note] InnoDB: Crash recovery did not find the parallel doublewrite buffer at /var/lib/mysql/xb_doublewrite
2025-02-21T04:57:33.080695Z 0 [Note] InnoDB: Highest supported file format is Barracuda.
2025-02-21T04:57:33.259714Z 0 [Note] InnoDB: Created parallel doublewrite buffer at /var/lib/mysql/xb_doublewrite, size 31457280 bytes
2025-02-21T04:57:33.499674Z 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2025-02-21T04:57:33.501311Z 0 [Note] InnoDB: Setting file ‘./ibtmp1’ size to 12 MB. Physically writing the file full; Please wait …
2025-02-21T04:57:33.531663Z 0 [Note] InnoDB: File ‘./ibtmp1’ size is now 12 MB.
2025-02-21T04:57:33.534446Z 0 [Note] InnoDB: 96 redo rollback segment(s) found. 96 redo rollback segment(s) are active.
2025-02-21T04:57:33.534471Z 0 [Note] InnoDB: 32 non-redo rollback segment(s) are active.
2025-02-21T04:57:33.534903Z 0 [Note] InnoDB: Waiting for purge to start
2025-02-21T04:57:33.586113Z 0 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.7.36-39 started; log sequence number 2759340
2025-02-21T04:57:33.586630Z 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
2025-02-21T04:57:33.586994Z 0 [Note] Plugin ‘FEDERATED’ is disabled.
2025-02-21T04:57:33.629425Z 0 [Note] Skipping generation of SSL certificates as options related to SSL are specified.
2025-02-21T04:57:33.629444Z 0 [Warning] A deprecated TLS version TLSv1 is enabled. Please use TLSv1.2 or higher.
2025-02-21T04:57:33.629446Z 0 [Warning] A deprecated TLS version TLSv1.1 is enabled. Please use TLSv1.2 or higher.
2025-02-21T04:57:33.630106Z 0 [Warning] CA certificate /etc/mysql/ssl-internal/ca.crt is self signed.
2025-02-21T04:57:33.630144Z 0 [Note] Skipping generation of RSA key pair as key files are present in data directory.
2025-02-21T04:57:33.635187Z 0 [Note] Server hostname (bind-address): ‘::’; port: 3306
2025-02-21T04:57:33.635226Z 0 [Note] - ‘::’ resolves to ‘::’;
2025-02-21T04:57:33.635267Z 0 [Note] Server socket created on IP: ‘::’.
2025-02-21T04:57:33.635302Z 0 [Note] Server hostname (bind-address): ‘::’; port: 33062
2025-02-21T04:57:33.635306Z 0 [Note] - ‘::’ resolves to ‘::’;
2025-02-21T04:57:33.635317Z 0 [Note] Server socket created on IP: ‘::’.
2025-02-21T04:57:33.636630Z 0 [Warning] Insecure configuration for --pid-file: Location ‘/var/lib/mysql’ in the path is accessible to all OS users. Consider choosing a different directory.
2025-02-21T04:57:33.658847Z 0 [Note] InnoDB: Buffer pool(s) load completed at 250221 4:57:33
2025-02-21T04:57:33.723845Z 0 [Note] Failed to start slave threads for channel ‘’
2025-02-21T04:57:33.910112Z 0 [Note] Event Scheduler: Loaded 0 events
2025-02-21T04:57:33.910581Z 0 [Note] mysqld: ready for connections.
Version: ‘5.7.36-39-57’ socket: ‘/tmp/mysql.sock’ port: 3306 Percona XtraDB Cluster (GPL), Release rel39, Revision 5197785, WSREP version 31.55, wsrep_31.55
2025-02-21T04:57:33.911345Z 2 [Note] WSREP: Initialized wsrep sidno 2
2025-02-21T04:57:33.911377Z 2 [Note] WSREP: Auto Increment Offset/Increment re-align with cluster membership change (Offset: 1 → 1) (Increment: 1 → 1)
2025-02-21T04:57:33.911440Z 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2025-02-21T04:57:33.911494Z 2 [Note] WSREP: Assign initial position for certification: 0, protocol version: 4
2025-02-21T04:57:33.911680Z 0 [Note] WSREP: Service thread queue flushed.
2025-02-21T04:57:33.912672Z 2 [Note] WSREP: Synchronized with group, ready for connections
2025-02-21T04:57:33.912719Z 2 [Note] WSREP: Setting wsrep_ready to true
2025-02-21T04:57:33.912730Z 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
–
Need help to identify and recover from this state.
