Unable to create Mongo cluster via percona mongodb operator

MongoDB Percona Operator for MongoDB
1

Description:

Mongodb cluster stays in the initialization stage for ever.

Steps to Reproduce:

apiVersion: psmdb.percona.com/v1
kind: PerconaServerMongoDB
metadata:
name: test-mongo
spec:
crVersion: 1.15.0
image: percona/percona-server-mongodb:6.0.9-7
allowUnsafeConfigurations: true
upgradeOptions:
apply: disabled
schedule: “0 2 * * *”
secrets:
users: minimal-cluster
replsets:

  • name: rs0
    size: 3
    volumeSpec:
    persistentVolumeClaim:
    resources:
    requests:
    storage: 3Gi

sharding:
enabled: true

configsvrReplSet:
  size: 3
  volumeSpec:
    persistentVolumeClaim:
      resources:
        requests:
          storage: 3Gi

mongos:
  size: 1

Logs:

Warning FailedScheduling 2m1s default-scheduler 0/1 nodes are available: pod has unbound immediate PersistentVolumeClaims. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling..
Warning FailedScheduling 2m default-scheduler 0/1 nodes are available: 1 node(s) didn’t match pod anti-affinity rules. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod..
[ec2-user@ip-172-31-35-114 percona-server-mongodb-operator]$

Actual Result:

NAME READY STATUS RESTARTS AGE
dbe-mongo-cfg-0 1/1 Running 1 (2m56s ago) 6m8s
dbe-mongo-cfg-1 0/1 Pending 0 5m36s
dbe-mongo-rs0-0 1/1 Running 1 (2m56s ago) 6m8s
dbe-mongo-rs0-1 0/1 Pending 0 5m36s
percona-server-mongodb-operator-6f87fc7889-mt7lk 1/1 Running 0 8m42s

2

Hi Sai Krishna,
How many nodes do you have for this setup? The error seems failing at allocating the nodes:

 Warning FailedScheduling 2m default-scheduler 0/1 nodes are available: 1 node(s) didn’t match pod anti-affinity rules. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod…

If you have a single node, then you will need to make sure to change the variable spec.replsets.affinity.antiAffinityTopologyKey / spec.sharding.configsvrReplSet.affinity.antiAffinityTopologyKey / spec.sharding.mongos.affinity.antiAffinityTopologyKey as follows and make sure you have enough resources in the node to allocate all pods:

    affinity:
      antiAffinityTopologyKey: "none"

If that is not the case, then you can provide us the below output to check the same (add namespace option if required):

kubectl get nodes
kubectl get psmdb
kubectl get events

Regards,
VK

3

Thanks Vinodh, It worked!

4

minukube cluster worked fine.

I’m also trying in EKS cluster. This is the error I’m seeing.

percona-server-mongodb-operator]$
NAME STATUS ROLES AGE VERSION
host1.ec2.internal Ready 29m v1.27.9-eks-5e0fdde
host2.internal Ready 30m v1.27.9-eks-5e0fdde
host3.ec2.internal Ready 30m v1.27.9-eks-5e0fdde

NAME ENDPOINT STATUS AGE
mongo-dev initializing 17m

Events:
Type Reason Age From Message

Warning FailedScheduling 4m53s default-scheduler running PreBind plugin “VolumeBinding”: binding volumes: timed out waiting for the condition

=====
inOkmgYNztfTEAql6CrIsn4x-88McrDfwaCVLCEcYxjH7G60pJ3F76bHGnwgfGpE-il_X2zlMbk1WJJSxcZgpkZGvFCecuAyIyk96tdmMfQXT5kuV56PFH_ufrOckJc7wfpSG03oe5dKWzKy3XPSWjxq-Pq0DO2CeWAi_HfqqgSyw8bicqe3R_sZQRfmmb514k2HRJ5gemkMBPj-rmXQe58-kbyRhqaEY-55N7AkFnr7SAMKxgEw
status code: 403, request id: 8e6e0196-82cc-4049-adc2-f681d8f313e5
11m Warning ProvisioningFailed persistentvolumeclaim/mongod-data-dbe-mongo-dev-rs0-0 failed to provision volume with StorageClass “gp2”: rpc error: code = Internal desc = Could not create volume “pvc-3fbf5aba-1ef6-4eab-9881-b3aae5d7d027”: could not create volume in EC2: UnauthorizedOperation: You are not authorized to perform this operation. User: arn:aws:sts::533267075098:assumed-role/eksctl-mongo-dev-nodegroup-standar-NodeInstanceRole-MVnMpxEAjld5/i-0a476e7918438676f is not authorized to perform: ec2:CreateVolume on resource: arn:aws:ec2:us-east-1:533267075098:volume/* because no identity-based policy allows the ec2:CreateVolume action. Encoded authorization failure message: 8OG9aEmYGMlHF4FGZxuCvZpl_mCtQ8sqrLetXdASueD_yAuFAc14Co8cLvwXF5s6oJpG4NCce5IqzOpunHbPIyvoVT__0tOY838DytDVXerZ6qFiTXt9ip3p-WZo5Bprw4iotjQytB7AaK2OlBrRUwvdYuyMdVoYdHfnZAmdYThEgi0J9pLmoKMXeZqIfLoBjbJ1GFyLLNke3gdBlZLJKIU_s9p06Jg9HmmWFrjPM8fMac5f0pjvGD2xwAXRz9m9V5RXMYdVQNTngPtAkREjoDg-eTLl6fEyqe21oM0aBfKHTd6-lwT_T8593alnNSTS4mb95j2v3BrjDx4ey3glHoFPE3ywKdlEsm_8tfuIVjAdP8jbivhZOGnPKc3JteOl1pXpRSVIeixHH0WShWJnC87KyMi-fBP8k1J_QczbD5Pygd73IwQTAcbuR0IT0jKBF1INt2jV65Tq5lh0Ie3ix3OetW-hsQYh5JAbAM7uRclSV17Z-GJ7tG0pJ9Exn3fYPsT9Hd4HWCmOlCRRh8a-7jdkymo2We5rJc0s6oWkBZfWS29fweEHON1BKS0M38TIyZbUYI9YA0-WXqCLJZDJoRtqy6heScZZ_n_yHfgAh9Nh6z6HcZTKV7_UtmVPcNa1yLZkZZaj8DAIvr9qQgIc8WgUw_dXdQ
status code: 403, request id: 06e01eb9-3ffa-4e57-be86-c062526eef7f
11m Warning ProvisioningFailed persistentvolumeclaim/mongod-data-dbe-mongo-dev-rs0-0 failed to provision volume with StorageClass “gp2”: rpc error: code = Internal desc = Could not create volume “pvc-3fbf5aba-1ef6-4eab-9881-b3aae5d7d027”: could not create volume in EC2: UnauthorizedOperation: You are not authorized to perform this operation. User: arn:aws:sts::533267075098:assumed-role/eksctl-mongo-dev-nodegroup-standar-NodeInstanceRole-MVnMpxEAjld5/i-0a476e7918438676f is not authorized to perform: ec2:CreateVolume on resource: arn:aws:ec2:us-east-1:533267075098:volume/* because no identity-based policy allows the ec2:CreateVolume action. Encoded authorization failure message: IhOmKw_jHsLUAOMXaJ6L6YtjXaFw_EF2gKOUrqOiZ-Ra6IwrM0Rl09rxNyb0jMtNEryBjqFm0jC-GBURoasbVPrtcl9RcBhFGhVTPZlN0I84fFhlAJMh6Beh7qxRN_UQC_XRk8psl6QI0ce0Uxw8HNuFIIw8WWyKcDDkwb7c8E_N2sJmly_d30j7QR97P0dgNi1OkBQlfZou-Oc_9auClAPI05xLKzWS85nP1UjkGPpqPzDHOLq3HkDNXOSURHrELQS7U4vlnEm69ZvMKi5qg90Val_GUD1T1OXdeggCqXq47WqGmoV3mV7TOrSO_pX5nAu04zagu5P0nrq0MAoUBnaV9CvAp68SFkMdc1I085TWi8Ln6cB4MoA58gwAmtZgx9CAVzH3-oEvcoPoWo7j8oXoUwFISO5KbIE3zrGH45AgIB6kjWbbL2-jHCosrrflVLqWpO4ZNXp81hroFGL6xDPGdwRe1WGWCwaytBCjjD0-UwFzro-7dKYIo0X9ZgVz6988Y6tl3NhtOlji4oGMKmUPzpF0tQfakBijnES6ashiogH8UM34IAHyqWBQx0eASKu-J37VsmjYNJ7xxT_yeByGZ_fGIg2xg99jGnDUqmH4ehHfpnAypSRU1XqSfa2Nsq6JXFjECGoQZCE9RYBrfcokPy4_Mg
status code: 403, request id: bb1753f5-7d85-4856-ba78-368c25d36559
10m Warning ProvisioningFailed persistentvolumeclaim/mongod-data-dbe-mongo-dev-rs0-0 failed to provision volume with StorageClass “gp2”: rpc error: code = Internal desc = Could not create volume “pvc-3fbf5aba-1ef6-4eab-9881-b3aae5d7d027”: could not create volume in EC2: UnauthorizedOperation: You are not authorized to perform this operation. User: arn:aws:sts::533267075098:assumed-role/eksctl-mongo-dev-nodegroup-standar-NodeInstanceRole-MVnMpxEAjld5/i-0a476e7918438676f is not authorized to perform: ec2:CreateVolume on resource: arn:aws:ec2:us-east-1:533267075098:volume/* because no identity-based policy allows the ec2:CreateVolume action. Encoded authorization failure message: T4aOtF8w1uprS4CCjp6I9YIQHVR85d8NsVwIwQHTReMxm7Wd0mDTN1VKudgQ63VyET5mElXWhhxUm4938ffSKSDYNK8yozi92fNYFrbr2fSPo8qbKQCF8lJZJJrEncBAZwpUvpH1U15SgD1a28jOjH_xWk6Wb0_pU39PDwpUriL6PUIMUJVyRaEYciRURkA7sdnPx5REjH1J2PpY04K4B688RGPLgvFjBhqXSP5lB4kbd_kaOBXXkAxY1WBtTOOYHTC_QmvNZwfMj0024uu0dEm24TOhBr8ORYv0gpRdpbk6t455UjBR2o1jSj-yarN-q7bRyYFvfmY47V7y1mvsPWfUpyFG2rt8iQ2nLSuOtpLgTvgO2VtHKfKDW51HG-uCY3rxmr4z_1WUfF9O_bntLXY2iboF3pkiuaPoQ9eda2CphMvzJJ6dUpTKBqYKaYoP4jge0k_h6_jl8BEObiL511pT_5_zCEjBI_-RgJRr0xIbmzi3cLUKaVhCoUSQvFPTP8kOLuWzwaXjqRLwNgAkWFbOp0Pr1xrFMcwh6EOOD8Wv-DlQlnxz4CwqGZ9ij5SqrFBUW-nVMlQdyW-_G9ZhmaiF7bxp0CkEJ5gk81fFqJQX81cXeCn7gYeTI9Ct-WFVZAjB0_F-tfl44nvCoy1ucQql4ClKuw
status code: 403, request id: 187feea9-ac70-45e1-a4de-a14374e801ef
9m24s Warning ProvisioningFailed persistentvolumeclaim/mongod-data-dbe-mongo-dev-rs0-0 failed to provision volume with StorageClass “gp2”: rpc error: code = Internal desc = Could not create volume “pvc-3fbf5aba-1ef6-4eab-9881-b3aae5d7d027”: could not create volume in EC2: UnauthorizedOperation: You are not authorized to perform this operation. User: arn:aws:sts::533267075098:assumed-role/eksctl-mongo-dev-nodegroup-standar-NodeInstanceRole-MVnMpxEAjld5/i-0a476e7918438676f is not authorized to perform: ec2:CreateVolume on resource: arn:aws:ec2:us-east-1:533267075098:volume/* because no identity-based policy allows the ec2:CreateVolume action. Encoded authorization failure message: KeyWw4Ft2JOmpFQ6PsoLZU-apE38l1VErxJuHNYeOrf6suio-oT7JNBUnP1T6Q4rt3OlQcR_kdNFrkBkfNIty0anpxkBjqFu1p8RMm70TMgeeBVcRurWS4jutSGd7gLj8FCayADFbVQX8S6tAiOByAKKPN-X6peflyNO8THwOSdr9shT6KaoJ_OLOMVntpFSs2WwApTDt-eScumGHcQ4JobIdNXOEkJpMtv-tmnooZ2pxwL6YWoab6-qfV7u17dF-WS5PbkwmUSidC118gQBaQQwSlk-6iGar2y-v2z9VCwHiD6FyHZddgzO1dN-xKmfFaVyFt5tYFOgKaYwXzNHO8MpQ_7gKFcc4wwxnNe7LC_RkhQXwLAdJNTMqVNonu47_HNW11LooRiCGOKAorTyWUeAlatGHOsqLEBNjGQQJPK5sG14S-wF5r0ep1yhqh8lCaXohiaEA5CBzLln6XPmWKh8i_iMQ_LpZcaSQlmmvPQiSk1FIS6m3hTkmaLVYZnkoY3ez4aDZY3iNb5TcDXUAuwxjurIoLZUNIcRtgIwjgjR7j4LssvX26TIT8QPA–fvwv1n02oukVrwU0J24xrzgi6edBOxXFgh55O7HqRNasuclJRZE293Ejh3rC5dFFecFkQTvm6sj5ycgrsH_XuD1L5UTtJ5A
status code: 403, request id: 4ad12c96-6d76-4ed7-8976-ace1843bb2e5
7m16s Warning ProvisioningFailed persistentvolumeclaim/mongod-data-dbe-mongo-dev-rs0-0 failed to provision volume with StorageClass “gp2”: rpc error: code = Internal desc = Could not create volume “pvc-3fbf5aba-1ef6-4eab-9881-b3aae5d7d027”: could not create volume in EC2: UnauthorizedOperation: You are not authorized to perform this operation. User: arn:aws:sts::533267075098:assumed-role/eksctl-mongo-dev-nodegroup-standar-NodeInstanceRole-MVnMpxEAjld5/i-0a476e7918438676f is not authorized to perform: ec2:CreateVolume on resource: arn:aws:ec2:us-east-1:533267075098:volume/* because no identity-based policy allows the ec2:CreateVolume action. Encoded authorization failure message: efKsVeEtXk47AFeEbx1age2pUv8EOfJzl3LpJWgeT6Hcjjp-Qw8HOvbSNd6mn4bQg5E5jRq0dGCYDnZl7SeYbgZNjukmIaSUzj6BZB5qS_BbYYhEyzBaMgIqQHv8_R5f72mQaZIDirWFkkpJgc_xLFxlY60VhWEHcJYieeo2k5_c2EMcIVqC0t4Z745l3QdwMGYaYmH6V14RYI_N_I215psqKNgaSI144zyd8ZdCpcTB8b-MJ10xVq3bn3ZN3nR1Mj3RO-so4mi1RTn71Z52oao1iXVEUcw-YeiH02drAV9UThq14I73gNTw3d3HhocLJgDVJrUOKKcTBPoSwxKGbZ7MzEmOfZ3oMP9PvH2kU5SMCGVPH4Ae7Y1nDNQFiNDHThf7Ls0gE9Va_F6n43lQKuqW_KWaePLk_rcbrtgRH5whWErBOQumU-sMtxsqoJjGK_hWURInWVrDROwcWq6tyTw4kzoJQqFT5CrKdNNBzyDNEEPShZ7m3hN9AkyfP5nZzWmI5yCmeG2ZwZfHeVCpRC4vBiRqpc90C5aUDLRd6H91FReDbXPXKtKmNcQpzvmI-dVnhdoACi-gzOQXeWh2slmUpnr_VFvJduiWQFYge3hHvU0dTtTJYa9DBztiRbDfMV2j5l13J-hY4Lx5DGTWRKknA-Bvnw
status code: 403, request id: 6a0700d3-4a5a-4b48-8d48-503ae0c0ffc1
3m Warning ProvisioningFailed persistentvolumeclaim/mongod-data-dbe-mongo-dev-rs0-0 (combined from similar events): failed to provision volume with StorageClass “gp2”: rpc error: code = Internal desc = Could not create volume “pvc-3fbf5aba-1ef6-4eab-9881-b3aae5d7d027”: could not create volume in EC2: UnauthorizedOperation: You are not authorized to perform this operation. User: arn:aws:sts::533267075098:assumed-role/eksctl-mongo-dev-nodegroup-standar-NodeInstanceRole-MVnMpxEAjld5/i-0a476e7918438676f is not authorized to perform: ec2:CreateVolume on resource: arn:aws:ec2:us-east-1:533267075098:volume/* because no identity-based policy allows the ec2:CreateVolume action. Encoded authorization failure message: 438GET8GM6r-G92YQZRE12ycbUaIbYeE7XN62CGq2yBFjUNdLjpo5k8yEPmWN640lTjeBvshzjAfZhzgtcDWmotRrd6rqGm1QUlBmm-HtPYYTzborO1BbFIn5k1ilPyaLV7w5roEuOFEp5k1WNI4nL_XEPk19IG0H_3aAWTc6sMCDS7nsyxXPqvP_lzeREnV0ZnlI9MWWroPjTr8n797UVhVTeNuS1KOTQyRwygEjgq6hWaGPpAsvCJT_Ha9rI8gVKJwp-FWNvAkkXKCbG2nd7zcArJC-VcXiuZXlVLzQbwpdxrNmo8FKJEsCJXUVSzXCYqhWlDb5e8Q94JRGJSosfhXG91IYWjUQI22HPX-19oMC4ZYSyv4XYIYPqPH0O0lx81xap66l3CFxIzp29TjCA9eNz_hxuaHV6FnuQJdxsMRNFJsYYByLtIzT-Eeq_s0WGO33eBj0hhSDgRhEFLIeCdvkMxVrnTJAzT7RWoze8JL-Ne5jLdoxdzW0w47qV1t605bfLH_Dss0MU0405zGIwphsc1uEtCbk3e3JqhVSgb_e5h8r1Q90uHSz4MpZiFVfq8qI-Ma3PAusMyeHXkzpGMSL3AoDj70NYT9n2Lya0t0ovWWi9qFfb55u7UxQpC3jXF5w-OmtYyvzOmSrP2duF5gG445fw
status code: 403, request id: 1e537990-6d3b-4f23-816b-da22877a187d

5

Hey Sai Krishna,
Glad that it worked. I suggest you open another new topic for the other question and solve this one.