Description:
Hi friends, i am attempting to deploy psmdb-db repleset in a new kubernetes aws eks 1.30 cluster. However the pod is stuck in “CrashLoopBackoff”.
The pod shows the follow error
exec /opt/percona/ps-entry.sh: exec format error
Steps to Reproduce:
psmdb-db-internal.yaml
finalizers:
- delete-psmdb-pods-in-order
- delete-psmdb-pvc
unsafeFlags:
replsetSize: true
replsets:
rs0:
name: rs0
size: 1
nodeSelector:
geeiq/node-type: mongodb
expose:
enabled: true
exposeType: LoadBalancer
serviceAnnotations:
service.beta.kubernetes.io/aws-load-balancer-target-group-attributes: preserve_client_ip.enabled=false
service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
service.beta.kubernetes.io/aws-load-balancer-scheme: internal
service.beta.kubernetes.io/aws-load-balancer-ip-address-type: dualstack
volumeSpec:
pvc:
storageClassName: mongodb
resources:
requests:
storage: 250Gi
backup:
enabled: false
sharding:
enabled: false
Version:
1.16.1
Logs:
kubectl logs -n mongodb psmdb-db-internal-rs0-0 mongod
exec /opt/percona/ps-entry.sh: exec format error
2m16s Warning BackOff pod/psmdb-db-internal-rs0-0 Back-off restarting failed container mongod in pod psmdb-db-internal-rs0-0_mongodb(72ca500a-9c2f-419c-9322-145836c27bef)
kubectl -n mongodb logs -f psmdb-operator-5457956884-wxsl9
2024-06-24T10:10:13.902Z ERROR failed to reconcile cluster {“controller”: “psmdb-controller”, “object”: {“name”:“psmdb-db-internal”,“namespace”:“mongodb”}, “namespace”: “mongodb”, “name”: “psmdb-db-internal”, “reconcileID”: “be4faeb2-8c95-4667-b7c5-f09ba30e37e8”, “replset”: “rs0”, “error”: “handleReplsetInit: no mongod containers in running state”, “errorVerbose”: “no mongod containers in running state\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.init\n\t:1\nruntime.doInit1\n\t/usr/local/go/src/runtime/proc.go:7176\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:7143\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:253\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_arm64.s:1222\nhandleReplsetInit\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).reconcileCluster\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/mgo.go:100\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).reconcileReplsets\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:551\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:402\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:222\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_arm64.s:1222”}
github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).reconcileReplsets
/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:553
github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).Reconcile
/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:402
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:114
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:311
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:261
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:222
Expected Result:
For the pod to be in Running status
Actual Result:
Pod Status CrashLoopBackoff
Additional Information:
k8s version: AWS EKS 1.30
IPV6 Only Cluster
db+operator version: 1.16.1
ec2 instance type: m7g.large (arm architecture)
kubectl get all -n mongodb
NAME READY STATUS RESTARTS AGE
pod/psmdb-db-internal-rs0-0 0/1 CrashLoopBackOff 5 (2m22s ago) 5m22s
pod/psmdb-operator-5457956884-wxsl9 1/1 Running 0 5m32s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/psmdb-db-internal-rs0 ClusterIP None <none> 27017/TCP 5m22s
service/psmdb-db-internal-rs0-0 LoadBalancer fdd1:fe9:30d1::72f0 k8s-mongodb-psmdbdbi-384d7c132b-fee5b14ed0ad0767.elb.eu-west-2.amazonaws.com 27017:31786/TCP 5m8s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/psmdb-operator 1/1 1 1 5m32s
NAME DESIRED CURRENT READY AGE
replicaset.apps/psmdb-operator-5457956884 1 1 1 5m32s
NAME READY AGE
statefulset.apps/psmdb-db-internal-rs0 0/1 5m22s
- I assumed there maybe some issue with my psmdb-db config yaml. I tried the quickstart guide installation from scratch. With Helm - Percona Operator for MongoDB however that errors aswell with the same error message
update
Update:
I switched from the aws graviton arm ec2 instance (m7g.large) to the (m5.large) and this error went away and the pod successfully Running. I would like to get this working with the arm graviton instances, is this a problem currently known?
- Liveness probe is now failing because causes crashloop ( different from the origianl post crashloop?)
Please let me know if you require any more information from me to help me.
psmdb-db logs
{"t":{"$date":"2024-06-24T13:04:28.470+00:00"},"s":"W", "c":"NETWORK", "id":21207, "ctx":"conn7","msg":"getaddrinfo() failed","attr":{"host":"psmdb-db-internal-rs0-0.psmdb-db-internal-rs0.mongodb.svc.cluster.local","error":"Name or service not known","timedOut":false}}
{"t":{"$date":"2024-06-24T13:04:28.472+00:00"},"s":"I", "c":"NETWORK", "id":4834700, "ctx":"conn7","msg":"isSelf could not connect via connectSocketOnly","attr":{"hostAndPort":"psmdb-db-internal-rs0-0.psmdb-db-internal-rs0.mongodb.svc.cluster.local:27017","error":{"code":6,"codeName":"HostUnreachable","errmsg":"couldn't connect to server psmdb-db-internal-rs0-0.psmdb-db-internal-rs0.mongodb.svc.cluster.local:27017, connection attempt failed: HostNotFound: Could not find address for psmdb-db-internal-rs0-0.psmdb-db-internal-rs0.mongodb.svc.cluster.local:27017: SocketException: resolve :: caused by :: Host not found (authoritative)"}}}
{"t":{"$date":"2024-06-24T13:04:28.473+00:00"},"s":"E", "c":"REPL", "id":21425, "ctx":"conn7","msg":"replSetInitiate error while validating config","attr":{"error":{"code":74,"codeName":"NodeNotFound","errmsg":"No host described in new configuration with {version: 1, term: 0} for replica set rs0 maps to this node"},"config":{"_id":"rs0","version":1,"members":[{"_id":0,"host":"psmdb-db-internal-rs0-0.psmdb-db-internal-rs0.mongodb.svc.cluster.local:27017"}]}}}
{"t":{"$date":"2024-06-24T13:04:28.473+00:00"},"s":"I", "c":"REPL", "id":6015317, "ctx":"conn7","msg":"Setting new configuration state","attr":{"newState":"ConfigUninitialized","oldState":"ConfigInitiating"}}
{"t":{"$date":"2024-06-24T13:04:28.558+00:00"},"s":"I", "c":"-", "id":20883, "ctx":"conn6","msg":"Interrupted operation as its client disconnected","attr":{"opId":2524}}
{"t":{"$date":"2024-06-24T13:04:29.001+00:00"},"s":"W", "c":"QUERY", "id":23799, "ctx":"ftdc","msg":"Aggregate command executor error","attr":{"error":{"code":26,"codeName":"NamespaceNotFound","errmsg":"Unable to retrieve storageStats in $collStats stage :: caused by :: Collection [local.oplog.rs] not found."},"stats":{},"cmd":{"aggregate":"oplog.rs","cursor":{},"pipeline":[{"$collStats":{"storageStats":{"waitForLock":false,"numericOnly":true}}}],"$db":"local"}}}
operator logs
2024-06-24T13:07:03.929Z ERROR failed to reconcile cluster {“controller”: “psmdb-controller”, “object”: {“name”:“psmdb-db-internal”,“namespace”:“mongodb”}, “namespace”: “mongodb”, “name”: “psmdb-db-internal”, “reconcileID”: “a19b8438-ae4e-415a-906e-87f2aafb487d”, “replset”: “rs0”, “error”: “handleReplsetInit: no mongod containers in running state”, “errorVerbose”: “no mongod containers in running state\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.init\n\t:1\nruntime.doInit1\n\t/usr/local/go/src/runtime/proc.go:7176\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:7143\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:253\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_arm64.s:1222\nhandleReplsetInit\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).reconcileCluster\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/mgo.go:100\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).reconcileReplsets\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:551\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:402\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:222\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_arm64.s:1222”}
github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).reconcileReplsets
/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:553
github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).Reconcile
/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:402
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:114
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:311
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:261
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:222
^C
Update 2
Liveness Probe issue went away after i enabld mongodb ipv6 support. So i have a configuration working properly on m5.large
configuration: |
net:
ipv6: true
So now im currently still facing the original issue, when using aws graviton arm ec2 instances, the psmdb-db pods fail with the folloiwng error: exec /opt/percona/ps-entry.sh: exec format error Ive trial and errored it a few times and confirmed it each time
- This looks interesting node.js - How to solve Docker container exec format error for running entrypoint - Stack Overflow
The exec format error message means that you have built your docker image on an ARM system like a Mac M1, and are trying to run it on an X86 system, or the opposite, building it on an X86 computer and trying to run it on an ARM system like an AWS Graviton environment.
The percona operator pod successfully runs on an arm ec2 instance but not the percona-server-mongodb and i can see from the above link there is no specific digest for the arm version of the server but there is for the operator.