GKE, maintenance, Volume is already exclusively attached to one node and can't be attached to another

MongoDB Percona Operator for MongoDB
1

Hello

After a GKE maintenance, some pods with PVs did not startup and “kubectl describe” returned:

Warning FailedAttachVolume 75s attachdetach-controller Multi-Attach error for volume “pvc-91f7dbad-4f50-47fc-9e09-68610309c779” Volume is already exclusively attached to one node and can’t be attached to another

Seems a node which got updated (cordoned) still has the mount.

Any workaround (ReadWriteOnce vs ReadWriteMany) to get around this issue ?

GKE 1.28, Operator 1.15

2

Hey @jamoser ,

seems like a storage interface (CSI) bug, or the volume is just stuck.
Don’t use ReadWriteMany - it might lead to various unexpected issues.
Find a way to unmount the volume and get it back.
If not possible (volume is corrupted/stuck/etc) - seems it calls for manual recovery - need to kill the Pod, delete the PV/PVC and allow MongoDB to recover the data for a newly provisioned replica set node.