# Multi-zone setup failure, if one zone is not avail **URL:** \ **Category:** Percona Operator for MongoDB\ **Created:** [October 6, 2025, 2:11pm UTC](https://forums.percona.com/t/multi-zone-setup-failure-if-one-zone-is-not-avail/39503 "2025-10-06T14:11:14Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![jamoser](https://avatars.discourse-cdn.com/v4/letter/j/b2d939/32.png) [@jamoser](https://forums.percona.com/u/jamoser)\ **Post date:** [October 6, 2025, 2:11pm UTC](https://forums.percona.com/t/multi-zone-setup-failure-if-one-zone-is-not-avail/39503/1 "2025-10-06T14:11:14Z")
Hello GKE: We set up our node pool such, that it has nodes from zone-a,b and c. When the mongodb pods first start, the disks are allocated from the respective zones → pvc from zone a, b and c. But if for example zone a is not avail and a restart is performed, the whole cluster will not start anymore. Seems the operator needs all pods be ready in order to fully start. Is this intended or bug ? Regards, John ---
**Author:** ![Ivan\_Groenewold](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/ivan_groenewold/32/6299_2.png) [@Ivan\_Groenewold](https://forums.percona.com/u/Ivan_Groenewold)\ **Post date:** [October 6, 2025, 3:12pm UTC](https://forums.percona.com/t/multi-zone-setup-failure-if-one-zone-is-not-avail/39503/2 "2025-10-06T15:12:17Z")
Hi John, that should not happen as far as I can tell. Can you explain exactly the steps you are following, your configuration yml and what do you see on operator pod logs? ---
**Author:** ![jamoser](https://avatars.discourse-cdn.com/v4/letter/j/b2d939/32.png) [@jamoser](https://forums.percona.com/u/jamoser)\ **Post date:** [October 8, 2025, 4:25pm UTC](https://forums.percona.com/t/multi-zone-setup-failure-if-one-zone-is-not-avail/39503/3 "2025-10-08T16:25:51Z")
Hello Ivan, Give me some time - but in the meantime: zone a was not available and pod for zone a tried to start. Since the disk is located on zone a and was missing, it just got stuck. the other pods did not have a chance to startup at all Regards, John ---
**Author:** ![Ege\_Gunes](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/ege_gunes/32/4595_2.png) [@Ege\_Gunes](https://forums.percona.com/u/Ege_Gunes)\ **Post date:** [October 9, 2025, 7:09am UTC](https://forums.percona.com/t/multi-zone-setup-failure-if-one-zone-is-not-avail/39503/4 "2025-10-09T07:09:22Z")
Hi @jamoser When StatefulSets are first created, pods are started in order. So pod-1 requires pod-0 to be up and running and all containers ready before it can start. If a StatefulSet is updated and requires a rolling restart, same rule applies but in reverse order. pod-2 will be restarted and pod-1 will require pod-2 to be ready. If you use `updateStrategy: SmartUpdate` in your cr.yaml, same rule applies but the order is determined by the pod’s role in replset. Secondaries are updated first and primary last. Rollout can be stuck because a pod might be crashing or pending. In this case a human needs to check the reason of crash or scheduling problem and fix. Otherwise a faulty config will be applied to all pods and the whole database cluster will be unavailable. ---
**Author:** ![jamoser](https://avatars.discourse-cdn.com/v4/letter/j/b2d939/32.png) [@jamoser](https://forums.percona.com/u/jamoser)\ **Post date:** [October 9, 2025, 7:53am UTC](https://forums.percona.com/t/multi-zone-setup-failure-if-one-zone-is-not-avail/39503/5 "2025-10-09T07:53:36Z")
Hi @Ege_Gunes lets say the pod 0 → zone a, pod 1 → zone b, pod 2 → zone c if zone a is gone, that means also disk 0 is gone, then pod 0 will not be able to startup. Therefore the whole cluster is not able to startup. I am not sure but with the setup below you have like redundancy, but it’s totally not transparent which zones are covered. parameters: type: pd-balanced replication-type: regional-pd ← where are the disk located ? Bottom line - as far as I understand - with the current setup it’s difficult to have a bullet proof redundant system. [edit: statefulset … consider this podManagementPolicy: Parallel … this should allow the statefulset start up even if one/some pod can not. but at least then you could resize the replicas and get quorum again. ] ---
**Author:** ![Slava\_Sarzhan](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/slava_sarzhan/32/3196_2.png) [@Slava\_Sarzhan](https://forums.percona.com/u/Slava_Sarzhan)\ **Post date:** [October 10, 2025, 10:33am UTC](https://forums.percona.com/t/multi-zone-setup-failure-if-one-zone-is-not-avail/39503/6 "2025-10-10T10:33:30Z")
Hi @jamoser, I have tested the following deployment. 1. I have created GKE k8s cluster ```auto ❯ kubectl get nodes -L topology.kubernetes.io/zone NAME STATUS ROLES AGE VERSION ZONE gke-slava-pxc-default-pool-0ea949fc-7jfz Ready 18m v1.33.5-gke.1125000 europe-west1-c gke-slava-pxc-default-pool-0ea949fc-hm8m Ready 18m v1.33.5-gke.1125000 europe-west1-c gke-slava-pxc-default-pool-0ea949fc-wrh8 Ready 18m v1.33.5-gke.1125000 europe-west1-c gke-slava-pxc-default-pool-5f9e79c2-dchk Ready 18m v1.33.5-gke.1125000 europe-west1-d gke-slava-pxc-default-pool-5f9e79c2-lnmc Ready 18m v1.33.5-gke.1125000 europe-west1-d gke-slava-pxc-default-pool-5f9e79c2-s8lc Ready 18m v1.33.5-gke.1125000 europe-west1-d gke-slava-pxc-default-pool-6b2df405-g1s0 Ready 18m v1.33.5-gke.1125000 europe-west1-b gke-slava-pxc-default-pool-6b2df405-l0vq Ready 18m v1.33.5-gke.1125000 europe-west1-b gke-slava-pxc-default-pool-6b2df405-m78v Ready 18m v1.33.5-gke.1125000 europe-west1-b ``` 1. Created SC: ```yaml apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: regional-fast provisioner: pd.csi.storage.gke.io parameters: type: pd-ssd # or pd-balanced for lower cost replication-type: regional-pd # KEY: Enables cross-zone replication volumeBindingMode: WaitForFirstConsumer allowVolumeExpansion: true reclaimPolicy: Delete allowedTopologies: - matchLabelExpressions: - key: topology.gke.io/zone values: - europe-west1-b - europe-west1-c - europe-west1-d ``` 1. Deployed operator and cluster using the following CR: ```yaml apiVersion: psmdb.percona.com/v1 kind: PerconaServerMongoDB metadata: name: my-cluster-name finalizers: - percona.com/delete-psmdb-pods-in-order spec: crVersion: "1.21.0" image: percona/percona-server-mongodb:7.0.14-8 replsets: - name: rs0 size: 3 # ZONE FAILOVER CONFIGURATION affinity: advanced: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - weight: 100 # High priority for zone spreading podAffinityTerm: labelSelector: matchLabels: app.kubernetes.io/name: percona-server-mongodb app.kubernetes.io/instance: my-cluster-name app.kubernetes.io/replset: rs0 topologyKey: topology.kubernetes.io/zone # USE REGIONAL STORAGE volumeSpec: persistentVolumeClaim: storageClassName: regional-fast resources: requests: storage: 10Gi # MAINTAIN AVAILABILITY DURING UPDATES podDisruptionBudget: maxUnavailable: 1 # RESOURCES resources: limits: cpu: "1" memory: 2Gi requests: cpu: 500m memory: 1Gi ``` 1. Removed one zone from node-pools ```yaml gcloud container node-pools update default-pool --cluster=*** -region=europe-west1 --node-locations=europe-west1-b,europe-west1-c ``` And as I can see, the pod was successfully rescheduled on a different node. Do you have any problems with this scenario? ---
**Author:** ![jamoser](https://avatars.discourse-cdn.com/v4/letter/j/b2d939/32.png) [@jamoser](https://forums.percona.com/u/jamoser)\ **Post date:** [October 14, 2025, 9:38am UTC](https://forums.percona.com/t/multi-zone-setup-failure-if-one-zone-is-not-avail/39503/7 "2025-10-14T09:38:15Z")
Hello @Slava_Sarzhan Yes - this works because you have basically 3 disk for each pod. So for 3 replicas you have to pay for 9 disks. But isn’t the idea that you have 3 pods each with 1 disk (each in a different zone) and since likely only 1 zone fails, the others should still be able to start ? Regards, John