Unable to start mongodb operator on EKS

MongoDB Percona Operator for MongoDB
1

Hi Team, I’m trying to launch mongo cluster via percona operator on EKS cluster.

percona-server-mongodb-operator]$
NAME STATUS ROLES AGE VERSION
host1.ec2.internal Ready 29m v1.27.9-eks-5e0fdde
host2.internal Ready 30m v1.27.9-eks-5e0fdde
host3.ec2.internal Ready 30m v1.27.9-eks-5e0fdde

NAME ENDPOINT STATUS AGE
mongo-dev initializing 17m

Events:
Type Reason Age From Message

---

Warning FailedScheduling 4m53s default-scheduler running PreBind plugin “VolumeBinding”: binding volumes: timed out waiting for the condition

=====
inOkmgYNztfTEAql6CrIsn4x-88McrDfwaCVLCEcYxjH7G60pJ3F76bHGnwgfGpE-il_X2zlMbk1WJJSxcZgpkZGvFCecuAyIyk96tdmMfQXT5kuV56PFH_ufrOckJc7wfpSG03oe5dKWzKy3XPSWjxq-Pq0DO2CeWAi_HfqqgSyw8bicqe3R_sZQRfmmb514k2HRJ5gemkMBPj-rmXQe58-kbyRhqaEY-55N7AkFnr7SAMKxgEw
status code: 403, request id: 8e6e0196-82cc-4049-adc2-f681d8f313e5
11m Warning ProvisioningFailed persistentvolumeclaim/mongod-data-dbe-mongo-dev-rs0-0 failed to provision volume with StorageClass “gp2”: rpc error: code = Internal desc = Could not create volume “pvc-3fbf5aba-1ef6-4eab-9881-b3aae5d7d027”: could not create volume in EC2: UnauthorizedOperation: You are not authorized to perform this operation. User: arn:aws:sts::533267075098:assumed-role/eksctl-mongo-dev-nodegroup-standar-NodeInstanceRole-MVnMpxEAjld5/i-0a476e7918438676f is not authorized to perform: ec2:CreateVolume on resource: arn:aws:ec2:us-east-1:533267075098:volume/* because no identity-based policy allows the ec2:CreateVolume action. Encoded authorization failure message: 8OG9aEmYGMlHF4FGZxuCvZpl_mCtQ8sqrLetXdASueD_yAuFAc14Co8cLvwXF5s6oJpG4NCce5IqzOpunHbPIyvoVT__0tOY838DytDVXerZ6qFiTXt9ip3p-WZo5Bprw4iotjQytB7AaK2OlBrRUwvdYuyMdVoYdHfnZAmdYThEgi0J9pLmoKMXeZqIfLoBjbJ1GFyLLNke3gdBlZLJKIU_s9p06Jg9HmmWFrjPM8fMac5f0pjvGD2xwAXRz9m9V5RXMYdVQNTngPtAkREjoDg-eTLl6fEyqe21oM0aBfKHTd6-lwT_T8593alnNSTS4mb95j2v3BrjDx4ey3glHoFPE3ywKdlEsm_8tfuIVjAdP8jbivhZOGnPKc3JteOl1pXpRSVIeixHH0WShWJnC87KyMi-fBP8k1J_QczbD5Pygd73IwQTAcbuR0IT0jKBF1INt2jV65Tq5lh0Ie3ix3OetW-hsQYh5JAbAM7uRclSV17Z-GJ7tG0pJ9Exn3fYPsT9Hd4HWCmOlCRRh8a-7jdkymo2We5rJc0s6oWkBZfWS29fweEHON1BKS0M38TIyZbUYI9YA0-WXqCLJZDJoRtqy6heScZZ_n_yHfgAh9Nh6z6HcZTKV7_UtmVPcNa1yLZkZZaj8DAIvr9qQgIc8WgUw_dXdQ
status code: 403, request id: 06e01eb9-3ffa-4e57-be86-c062526eef7f
11m Warning ProvisioningFailed persistentvolumeclaim/mongod-data-dbe-mongo-dev-rs0-0 failed to provision volume with StorageClass “gp2”: rpc error: code = Internal desc = Could not create volume “pvc-3fbf5aba-1ef6-4eab-9881-b3aae5d7d027”: could not create volume in EC2: UnauthorizedOperation: You are not authorized to perform this operation. User: arn:aws:sts::533267075098:assumed-role/eksctl-mongo-dev-nodegroup-standar-NodeInstanceRole-MVnMpxEAjld5/i-0a476e7918438676f is not authorized to perform: ec2:CreateVolume on resource: arn:aws:ec2:us-east-1:533267075098:volume/* because no identity-based policy allows the ec2:CreateVolume action. Encoded authorization failure message: IhOmKw_jHsLUAOMXaJ6L6YtjXaFw_EF2gKOUrqOiZ-Ra6IwrM0Rl09rxNyb0jMtNEryBjqFm0jC-GBURoasbVPrtcl9RcBhFGhVTPZlN0I84fFhlAJMh6Beh7qxRN_UQC_XRk8psl6QI0ce0Uxw8HNuFIIw8WWyKcDDkwb7c8E_N2sJmly_d30j7QR97P0dgNi1OkBQlfZou-Oc_9auClAPI05xLKzWS85nP1UjkGPpqPzDHOLq3HkDNXOSURHrELQS7U4vlnEm69ZvMKi5qg90Val_GUD1T1OXdeggCqXq47WqGmoV3mV7TOrSO_pX5nAu04zagu5P0nrq0MAoUBnaV9CvAp68SFkMdc1I085TWi8Ln6cB4MoA58gwAmtZgx9CAVzH3-oEvcoPoWo7j8oXoUwFISO5KbIE3zrGH45AgIB6kjWbbL2-jHCosrrflVLqWpO4ZNXp81hroFGL6xDPGdwRe1WGWCwaytBCjjD0-UwFzro-7dKYIo0X9ZgVz6988Y6tl3NhtOlji4oGMKmUPzpF0tQfakBijnES6ashiogH8UM34IAHyqWBQx0eASKu-J37VsmjYNJ7xxT_yeByGZ_fGIg2xg99jGnDUqmH4ehHfpnAypSRU1XqSfa2Nsq6JXFjECGoQZCE9RYBrfcokPy4_Mg
status code: 403, request id: bb1753f5-7d85-4856-ba78-368c25d36559
10m Warning ProvisioningFailed persistentvolumeclaim/mongod-data-dbe-mongo-dev-rs0-0 failed to provision volume with StorageClass “gp2”: rpc error: code = Internal desc = Could not create volume “pvc-3fbf5aba-1ef6-4eab-9881-b3aae5d7d027”: could not create volume in EC2: UnauthorizedOperation: You are not authorized to perform this operation. User: arn:aws:sts::533267075098:assumed-role/eksctl-mongo-dev-nodegroup-standar-NodeInstanceRole-MVnMpxEAjld5/i-0a476e7918438676f is not authorized to perform: ec2:CreateVolume on resource: arn:aws:ec2:us-east-1:533267075098:volume/* because no identity-based policy allows the ec2:CreateVolume action. Encoded authorization failure message: T4aOtF8w1uprS4CCjp6I9YIQHVR85d8NsVwIwQHTReMxm7Wd0mDTN1VKudgQ63VyET5mElXWhhxUm4938ffSKSDYNK8yozi92fNYFrbr2fSPo8qbKQCF8lJZJJrEncBAZwpUvpH1U15SgD1a28jOjH_xWk6Wb0_pU39PDwpUriL6PUIMUJVyRaEYciRURkA7sdnPx5REjH1J2PpY04K4B688RGPLgvFjBhqXSP5lB4kbd_kaOBXXkAxY1WBtTOOYHTC_QmvNZwfMj0024uu0dEm24TOhBr8ORYv0gpRdpbk6t455UjBR2o1jSj-yarN-q7bRyYFvfmY47V7y1mvsPWfUpyFG2rt8iQ2nLSuOtpLgTvgO2VtHKfKDW51HG-uCY3rxmr4z_1WUfF9O_bntLXY2iboF3pkiuaPoQ9eda2CphMvzJJ6dUpTKBqYKaYoP4jge0k_h6_jl8BEObiL511pT_5_zCEjBI_-RgJRr0xIbmzi3cLUKaVhCoUSQvFPTP8kOLuWzwaXjqRLwNgAkWFbOp0Pr1xrFMcwh6EOOD8Wv-DlQlnxz4CwqGZ9ij5SqrFBUW-nVMlQdyW-_G9ZhmaiF7bxp0CkEJ5gk81fFqJQX81cXeCn7gYeTI9Ct-WFVZAjB0_F-tfl44nvCoy1ucQql4ClKuw
status code: 403, request id: 187feea9-ac70-45e1-a4de-a14374e801ef
9m24s Warning ProvisioningFailed persistentvolumeclaim/mongod-data-dbe-mongo-dev-rs0-0 failed to provision volume with StorageClass “gp2”: rpc error: code = Internal desc = Could not create volume “pvc-3fbf5aba-1ef6-4eab-9881-b3aae5d7d027”: could not create volume in EC2: UnauthorizedOperation: You are not authorized to perform this operation. User: arn:aws:sts::533267075098:assumed-role/eksctl-mongo-dev-nodegroup-standar-NodeInstanceRole-MVnMpxEAjld5/i-0a476e7918438676f is not authorized to perform: ec2:CreateVolume on resource: arn:aws:ec2:us-east-1:533267075098:volume/* because no identity-based policy allows the ec2:CreateVolume action. Encoded authorization failure message: KeyWw4Ft2JOmpFQ6PsoLZU-apE38l1VErxJuHNYeOrf6suio-oT7JNBUnP1T6Q4rt3OlQcR_kdNFrkBkfNIty0anpxkBjqFu1p8RMm70TMgeeBVcRurWS4jutSGd7gLj8FCayADFbVQX8S6tAiOByAKKPN-X6peflyNO8THwOSdr9shT6KaoJ_OLOMVntpFSs2WwApTDt-eScumGHcQ4JobIdNXOEkJpMtv-tmnooZ2pxwL6YWoab6-qfV7u17dF-WS5PbkwmUSidC118gQBaQQwSlk-6iGar2y-v2z9VCwHiD6FyHZddgzO1dN-xKmfFaVyFt5tYFOgKaYwXzNHO8MpQ_7gKFcc4wwxnNe7LC_RkhQXwLAdJNTMqVNonu47_HNW11LooRiCGOKAorTyWUeAlatGHOsqLEBNjGQQJPK5sG14S-wF5r0ep1yhqh8lCaXohiaEA5CBzLln6XPmWKh8i_iMQ_LpZcaSQlmmvPQiSk1FIS6m3hTkmaLVYZnkoY3ez4aDZY3iNb5TcDXUAuwxjurIoLZUNIcRtgIwjgjR7j4LssvX26TIT8QPA–fvwv1n02oukVrwU0J24xrzgi6edBOxXFgh55O7HqRNasuclJRZE293Ejh3rC5dFFecFkQTvm6sj5ycgrsH_XuD1L5UTtJ5A
status code: 403, request id: 4ad12c96-6d76-4ed7-8976-ace1843bb2e5
7m16s Warning ProvisioningFailed persistentvolumeclaim/mongod-data-dbe-mongo-dev-rs0-0 failed to provision volume with StorageClass “gp2”: rpc error: code = Internal desc = Could not create volume “pvc-3fbf5aba-1ef6-4eab-9881-b3aae5d7d027”: could not create volume in EC2: UnauthorizedOperation: You are not authorized to perform this operation. User: arn:aws:sts::533267075098:assumed-role/eksctl-mongo-dev-nodegroup-standar-NodeInstanceRole-MVnMpxEAjld5/i-0a476e7918438676f is not authorized to perform: ec2:CreateVolume on resource: arn:aws:ec2:us-east-1:533267075098:volume/* because no identity-based policy allows the ec2:CreateVolume action. Encoded authorization failure message: efKsVeEtXk47AFeEbx1age2pUv8EOfJzl3LpJWgeT6Hcjjp-Qw8HOvbSNd6mn4bQg5E5jRq0dGCYDnZl7SeYbgZNjukmIaSUzj6BZB5qS_BbYYhEyzBaMgIqQHv8_R5f72mQaZIDirWFkkpJgc_xLFxlY60VhWEHcJYieeo2k5_c2EMcIVqC0t4Z745l3QdwMGYaYmH6V14RYI_N_I215psqKNgaSI144zyd8ZdCpcTB8b-MJ10xVq3bn3ZN3nR1Mj3RO-so4mi1RTn71Z52oao1iXVEUcw-YeiH02drAV9UThq14I73gNTw3d3HhocLJgDVJrUOKKcTBPoSwxKGbZ7MzEmOfZ3oMP9PvH2kU5SMCGVPH4Ae7Y1nDNQFiNDHThf7Ls0gE9Va_F6n43lQKuqW_KWaePLk_rcbrtgRH5whWErBOQumU-sMtxsqoJjGK_hWURInWVrDROwcWq6tyTw4kzoJQqFT5CrKdNNBzyDNEEPShZ7m3hN9AkyfP5nZzWmI5yCmeG2ZwZfHeVCpRC4vBiRqpc90C5aUDLRd6H91FReDbXPXKtKmNcQpzvmI-dVnhdoACi-gzOQXeWh2slmUpnr_VFvJduiWQFYge3hHvU0dTtTJYa9DBztiRbDfMV2j5l13J-hY4Lx5DGTWRKknA-Bvnw
status code: 403, request id: 6a0700d3-4a5a-4b48-8d48-503ae0c0ffc1
3m Warning ProvisioningFailed persistentvolumeclaim/mongod-data-dbe-mongo-dev-rs0-0 (combined from similar events): failed to provision volume with StorageClass “gp2”: rpc error: code = Internal desc = Could not create volume “pvc-3fbf5aba-1ef6-4eab-9881-b3aae5d7d027”: could not create volume in EC2: UnauthorizedOperation: You are not authorized to perform this operation. User: arn:aws:sts::533267075098:assumed-role/eksctl-mongo-dev-nodegroup-standar-NodeInstanceRole-MVnMpxEAjld5/i-0a476e7918438676f is not authorized to perform: ec2:CreateVolume on resource: arn:aws:ec2:us-east-1:533267075098:volume/* because no identity-based policy allows the ec2:CreateVolume action. Encoded authorization failure message: 438GET8GM6r-G92YQZRE12ycbUaIbYeE7XN62CGq2yBFjUNdLjpo5k8yEPmWN640lTjeBvshzjAfZhzgtcDWmotRrd6rqGm1QUlBmm-HtPYYTzborO1BbFIn5k1ilPyaLV7w5roEuOFEp5k1WNI4nL_XEPk19IG0H_3aAWTc6sMCDS7nsyxXPqvP_lzeREnV0ZnlI9MWWroPjTr8n797UVhVTeNuS1KOTQyRwygEjgq6hWaGPpAsvCJT_Ha9rI8gVKJwp-FWNvAkkXKCbG2nd7zcArJC-VcXiuZXlVLzQbwpdxrNmo8FKJEsCJXUVSzXCYqhWlDb5e8Q94JRGJSosfhXG91IYWjUQI22HPX-19oMC4ZYSyv4XYIYPqPH0O0lx81xap66l3CFxIzp29TjCA9eNz_hxuaHV6FnuQJdxsMRNFJsYYByLtIzT-Eeq_s0WGO33eBj0hhSDgRhEFLIeCdvkMxVrnTJAzT7RWoze8JL-Ne5jLdoxdzW0w47qV1t605bfLH_Dss0MU0405zGIwphsc1uEtCbk3e3JqhVSgb_e5h8r1Q90uHSz4MpZiFVfq8qI-Ma3PAusMyeHXkzpGMSL3AoDj70NYT9n2Lya0t0ovWWi9qFfb55u7UxQpC3jXF5w-OmtYyvzOmSrP2duF5gG445fw
status code: 403, request id: 1e537990-6d3b-4f23-816b-da22877a187d
2

Hello @Sai_krishna ,

there can be a few reasons for that:

  1. You did not install and (or) properly configured EBS container storage interface (CSI). Please have a look at this doc: Managing the Amazon EBS CSI driver as an Amazon EKS add-on - Amazon EKS
    Most likely you forgot RBAC : external-snapshotter/deploy/kubernetes/snapshot-controller/rbac-snapshot-controller.yaml at master · kubernetes-csi/external-snapshotter · GitHub

  2. Judging by the errors that you shared, it is something to do with permissions. So it is either RBAC again or some other permissions that your cluster lacks.

Most likely it is (1). Try out the doc that AWS shared for EBS CSI: Managing the Amazon EBS CSI driver as an Amazon EKS add-on - Amazon EKS
And let me know if you still have questions.

1 Like