Operator PS MySQL - behaviour on 1 instance that gets unavailable

Carlos_Abrantes · August 17, 2026, 10:36am

Hi all,

just trying to understand the expected operator behaviour.

Description:

a cluster with 3 instances using Group Replication

one pod/instance gets unavailable (network split 1 vs 2).
It gets the state UNREACHABLE
It gets removed from the cluster topology
the pod is available again

In this situation seems that a Full Cluster Crash is “issued” causing downtime, can someone confirm that this is the expected behaviour? or the system should recover without downtime?

I would expect that a single instance failure/recovery would not require downtime (maybe i m doing something wrong).

Version:

1.2.0

Thanks!

(edit)
What i checked with some AI help:

Seems that the minority instance in network split (1 vs 2) when is not able to reach any member writes the /var/lib/mysql/full-cluster-crash.

This happens even if 2 of 3 members are healthy and running fine.

The operator trusts blindly on the minority instance that does not see the other 2 instances and runs the rebootClusterFromCompleteOutage.

I guess in this case the operator could/should confirm the state of the other instances to check if the cluster is or not healthy/running.

anil.joshi · August 18, 2026, 5:16am

@Carlos_Abrantes

If one of the GR members is unreachable or down, the group should still remain intact. If the primary itself goes down, the Group Replication process triggers an automatic primary election, selecting one of the available members to become the new primary.

What was the Group Replication status among all active members?
select * from performance_schema.replication_group_members

Having group members’ pod-level logs and the mysqld error logs could provide better insight. Can you share those if feasible?

Carlos_Abrantes · August 18, 2026, 4:34pm

Hi @anil.joshi ,

Thanks for your time, i will try to detail what i did.

1- Cluster is healthy, all members are up and running.

init state

{
    "clusterName": "psdb",
    "defaultReplicaSet": {
        "name": "default",
        "primary": "ps-db-mysql-1.ps-db-mysql.ps-mysql:3306",
        "ssl": "REQUIRED",
        "status": "OK",
        "statusText": "Cluster is ONLINE and can tolerate up to ONE failure.",
        "topology": {
            "ps-db-mysql-0.ps-db-mysql.ps-mysql:3306": {
                "address": "ps-db-mysql-0.ps-db-mysql.ps-mysql:3306",
                "memberRole": "SECONDARY",
                "mode": "R/O",
                "readReplicas": {},
                "replicationLag": "applier_queue_applied",
                "role": "HA",
                "status": "ONLINE",
                "version": "8.4.10"
            },
            "ps-db-mysql-1.ps-db-mysql.ps-mysql:3306": {
                "address": "ps-db-mysql-1.ps-db-mysql.ps-mysql:3306",
                "memberRole": "PRIMARY",
                "mode": "R/W",
                "readReplicas": {},
                "replicationLag": "applier_queue_applied",
                "role": "HA",
                "status": "ONLINE",
                "version": "8.4.10"
            },
            "ps-db-mysql-2.ps-db-mysql.ps-mysql:3306": {
                "address": "ps-db-mysql-2.ps-db-mysql.ps-mysql:3306",
                "memberRole": "SECONDARY",
                "mode": "R/O",
                "readReplicas": {},
                "replicationLag": "applier_queue_applied",
                "role": "HA",
                "status": "ONLINE",
                "version": "8.4.10"
            }
        },
        "topologyMode": "Single-Primary"
    },
    "groupInformationSourceMember": "ps-db-mysql-1.ps-db-mysql.ps-mysql:3306"
}



+---------------------------+--------------------------------------+------------------------------------+-------------+--------------+-------------+----------------+----------------------------+
| CHANNEL_NAME              | MEMBER_ID                            | MEMBER_HOST                        | MEMBER_PORT | MEMBER_STATE | MEMBER_ROLE | MEMBER_VERSION | MEMBER_COMMUNICATION_STACK |
+---------------------------+--------------------------------------+------------------------------------+-------------+--------------+-------------+----------------+----------------------------+
| group_replication_applier | 2fec5bd7-9aed-11f1-bf8f-0e5e11966261 | ps-db-mysql-0.ps-db-mysql.ps-mysql |        3306 | ONLINE       | SECONDARY   | 8.4.10         | MySQL                      |
| group_replication_applier | 515fc4e2-9aed-11f1-b6ba-96a4c252d4b1 | ps-db-mysql-1.ps-db-mysql.ps-mysql |        3306 | ONLINE       | PRIMARY     | 8.4.10         | MySQL                      |
| group_replication_applier | 911ce1a2-9af6-11f1-89c6-368634cb2e7c | ps-db-mysql-2.ps-db-mysql.ps-mysql |        3306 | ONLINE       | SECONDARY   | 8.4.10         | MySQL                      |
+---------------------------+--------------------------------------+------------------------------------+-------------+--------------+-------------+----------------+----------------------------+

2- Then i deployed a network policy to simulate some network split.

network policy

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: partition-ps-db-mysql-0-from-peers
  namespace: ps-mysql
spec:
  podSelector:
    matchLabels:
      pod-name: ps-db-mysql-0

  policyTypes:
    - Ingress
    - Egress

  ingress:
    - from:
        - namespaceSelector: {}
          podSelector:
            matchExpressions:
              - key: pod-name
                operator: NotIn
                values:
                  - ps-db-mysql-1
                  - ps-db-mysql-2

        - ipBlock:
            cidr: 0.0.0.0/0

  egress:
    - to:
        - namespaceSelector: {}
          podSelector:
            matchExpressions:
              - key: pod-name
                operator: NotIn
                values:
                  - ps-db-mysql-1
                  - ps-db-mysql-2

        - ipBlock:
            cidr: 0.0.0.0/0

3- After the network policy we could see the expected cluster state, 1 node missing:

Cluster state after "network failure"

cluster.status()
{
    "clusterName": "psdb",
    "defaultReplicaSet": {
        "name": "default",
        "primary": "ps-db-mysql-1.ps-db-mysql.ps-mysql:3306",
        "ssl": "REQUIRED",
        "status": "OK_NO_TOLERANCE_PARTIAL",
        "statusText": "Cluster is NOT tolerant to any failures. 1 member is not active.",
        "topology": {
            "ps-db-mysql-0.ps-db-mysql.ps-mysql:3306": {
                "address": "ps-db-mysql-0.ps-db-mysql.ps-mysql:3306",
                "memberRole": "SECONDARY",
                "mode": "n/a",
                "readReplicas": {},
                "role": "HA",
                "shellConnectError": "MySQL Error 2003: Could not open connection to 'ps-db-mysql-0.ps-db-mysql.ps-mysql:3306': Can't connect to MySQL server on 'ps-db-mysql-0.ps-db-mysql.ps-mysql:3306' (110)",
                "status": "(MISSING)"
            },
            "ps-db-mysql-1.ps-db-mysql.ps-mysql:3306": {
                "address": "ps-db-mysql-1.ps-db-mysql.ps-mysql:3306",
                "memberRole": "PRIMARY",
                "mode": "R/W",
                "readReplicas": {},
                "replicationLag": "applier_queue_applied",
                "role": "HA",
                "status": "ONLINE",
                "version": "8.4.10"
            },
            "ps-db-mysql-2.ps-db-mysql.ps-mysql:3306": {
                "address": "ps-db-mysql-2.ps-db-mysql.ps-mysql:3306",
                "memberRole": "SECONDARY",
                "mode": "R/O",
                "readReplicas": {},
                "replicationLag": "applier_queue_applied",
                "role": "HA",
                "status": "ONLINE",
                "version": "8.4.10"
            }
        },
        "topologyMode": "Single-Primary"
    },
    "groupInformationSourceMember": "ps-db-mysql-1.ps-db-mysql.ps-mysql:3306"
}

Shortly after the “network failure” the pod gets not ready and the container is restarted.
The pod gets back without any intervention, meaning still in “network failure” to ready state.

In that pod we can find:

cat /var/lib/mysql/full-cluster-crash
2fec5bd7-9aed-11f1-bf8f-0e5e11966261:1-9,3d6d74ae-9aed-11f1-a0a7-0e5e11966261:1-137

4- Once the network policy is removed, “network failure is solved”

All the mysql pods get deleted, even if all the time until now the cluster was up and running with 2 instances as shown in the previous point.

In attach the logs from all instances and operator, one key log i found from the operator is:

Crash recovery	Tried to reboot the cluster but MySQL says the cluster is already online. Deleting all MySQL pods.

I tried to split the logs into moments, so expect to find some new lines with a comment.

Thanks

Carlos_Abrantes · August 18, 2026, 4:37pm

Seems i can’t attach the log files:

Sorry, new users can not upload attachments.

should i wait to have the permission or should just copy paste to a details tag?

Thanks

Ege_Gunes · August 19, 2026, 7:35am

Hi @Carlos_Abrantes,

I can confirm this behavior exists right now but it’s not the expected behavior for sure. We’re fixing this in v1.3.0 which should be released in September.

Carlos_Abrantes · August 19, 2026, 8:31am

Hi @Ege_Gunes

Thanks for the confirmation/update, i will wait for the new release and proceed with the tests then.

Thanks!

Topic		Replies	Views
Percona operator 0.8: Can't recover from OOM crash Percona Operator for MySQL	1	230	July 27, 2024
MySQL instance is in CrashLoopBackOff after killing pod Percona Operator for MySQL	1	334	September 8, 2025
Failover Problem with PS DB and Percona Operator Percona Operator for MySQL	6	1172	December 8, 2022
Momentarily Inconsistent group replication status during auto rejoin of a network partitioned member MySQL & MariaDB	10	692	February 13, 2024
Percona Operator for MySQL based on Percona xtradb cluster HA problems Percona Operator for MySQL	2	918	August 18, 2023

Operator PS MySQL - behaviour on 1 instance that gets unavailable

Description:

Version:

Related topics