Volume mount failed on iSCSI

MongoDB Percona Operator for MongoDB
1

Hello everyone. I tried to install Percona operator on k3s.
I edited the crd manifest to use a storage class I had previously created with Trident to interface with two ONTAP iSCSI disks. I had already verified that volumes pertaining to this storage class could be successfully mounted on ordinary pods.
Nevertheless, when I tried to deploy the modified crd manifest I got the following output.

  Warning  FailedMount  31m (x73 over 13h)         kubelet  Unable to attach or mount volumes: unmounted volumes=[mongod-data], unattached volumes=[mongod-data kube-api-access-9kwgq my-cluster-name-mongodb-keyfile ssl ssl-internal my-cluster-name-mongodb-encryption-key]: timed out waiting for the condition
  Warning  FailedMount  15m (x43 over 13h)         kubelet  Unable to attach or mount volumes: unmounted volumes=[mongod-data], unattached volumes=[ssl-internal my-cluster-name-mongodb-encryption-key mongod-data kube-api-access-9kwgq my-cluster-name-mongodb-keyfile ssl]: timed out waiting for the condition
  Warning  FailedMount  10m (x67 over 13h)         kubelet  Unable to attach or mount volumes: unmounted volumes=[mongod-data], unattached volumes=[my-cluster-name-mongodb-encryption-key mongod-data kube-api-access-9kwgq my-cluster-name-mongodb-keyfile ssl ssl-internal]: timed out waiting for the condition
  Warning  FailedMount  6m17s (x57 over 12h)       kubelet  Unable to attach or mount volumes: unmounted volumes=[mongod-data], unattached volumes=[kube-api-access-9kwgq my-cluster-name-mongodb-keyfile ssl ssl-internal my-cluster-name-mongodb-encryption-key mongod-data]: timed out waiting for the condition
  Warning  FailedMount  <invalid> (x247 over 13h)  kubelet  MountVolume.MountDevice failed for volume "pvc-4e91d600-7d68-43c9-b27a-461e85b7f444" : rpc error: code = Internal desc = no devices present yet

Have you got any clue about the reason why it’s happening?
Thank you a lot for your attention.

Other contextual information:

  • k3s v. 1.21.3
  • Trident v. 21.04
  • Percona operator v. 4.4.3-5
2

Hey @Gloria_Pedemonte ,

well, it says that storage device is not there. What do you see in kubectl get pvc ? Were volumes created already?

3

Thank you for your answer, @Sergey_Pronin .
Yes, the pvc has been created and bound. Moreover, if I run kubectl describe pvc it appears used by the first mongo pod.

4

Operator does not do much magic here. It provisions statefulset, statefulset has specific storage class for volume in the volume spec.

Does trident work for other pods?

6

It does work for other pods.

7

Hello @Gloria_Pedemonte

I don’t have trident env to reproduce.
We will be able to help if you share more details - Pods, PVCs, Statefulsets statuses, describes, yamls, etc.

Also please let me know what you have already tried and what you see.

8

Hello again @Sergey_Pronin,
sorry for the belated answer. I discovered this same problem extends to other stetefulsets, so it’s probably more like a trident-related issue.
I will post updates as soon as there are any.
Thank you for your time.