Operator deployment on OpenShift via helm

MongoDB Percona Operator for MongoDB
1

Description:

I am trying to install Percona Operator for MongoDB on a air-gapped OpenShift cluster. I cannot use the OperatorHub installation method unfortunately, so I fallback to installing with the helm chart.

The problem is that the operator Pod fails to start with this warning event:

Warning  Failed          17s               kubelet            Error: container create failed: time="2026-01-19T09:41:41Z" level=error msg="runc create failed: unable to start container process: exec: \\"percona-server-mongodb-operator\\": executable file not found in $PATH"

Further debug with a privileged container shows that in fact the /usr/local/bin directory seems to be empty. For that reason I don’t know whether this is an issue with the operator, OpenShift, or the image itself.

Steps to Reproduce:

helm install psmdb-operator -n psmdb-operator percona/psmdb-operator

Version:

Operator version: 1.21.1
Chart version: 1.21.2

Logs:

No logs other than the warning event above

Expected Result:

The operator installs successfully

Actual Result:

The container fails to start

2

Can you share kubectl describe pod <operator>? and also more logs from the pod?

3

There are no logs, as the container fails to start

Name:             psmdb-operator-6f995c5687-8ztzh
Namespace:        fi601-mcx-percona
Priority:         0
Service Account:  psmdb-operator
Node:             appworker0.fi-601.tre.nsn-rdnet.net/100.103.193.28
Start Time:       Mon, 19 Jan 2026 11:49:05 +0100
Labels:           app.kubernetes.io/instance=psmdb-operator
                  app.kubernetes.io/name=psmdb-operator
                  pod-template-hash=6f995c5687
Annotations:      k8s.ovn.org/pod-networks:
                    {"default":{"ip_addresses":["10.128.12.215/23"],"mac_address":"0a:58:0a:80:0c:d7","gateway_ips":["10.128.12.1"],"routes":[{"dest":"10.128....
                  k8s.v1.cni.cncf.io/network-status:
                    [{
                        "name": "ovn-kubernetes",
                        "interface": "eth0",
                        "ips": [
                            "10.128.12.215"
                        ],
                        "mac": "0a:58:0a:80:0c:d7",
                        "default": true,
                        "dns": {}
                    }]
                  openshift.io/scc: anyuid
Status:           Pending
IP:               10.128.12.215
IPs:
  IP:           10.128.12.215
Controlled By:  ReplicaSet/psmdb-operator-6f995c5687
Containers:
  psmdb-operator:
    Container ID:
    Image:         ncd-harbor.fi-601.tre.nsn-rdnet.net/mcx/percona/percona-server-mongodb-operator:1.21.1
    Image ID:
    Ports:         8080/TCP (metrics), 8081/TCP (health)
    Host Ports:    0/TCP (metrics), 0/TCP (health)
    Command:
      percona-server-mongodb-operator
    State:          Waiting
      Reason:       CreateContainerError
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:     100m
      memory:  128Mi
    Requests:
      cpu:      100m
      memory:   128Mi
    Liveness:   http-get http://:health/healthz delay=0s timeout=1s period=10s #success=1 #failure=3
    Readiness:  http-get http://:health/healthz delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:
      LOG_STRUCTURED:             false
      LOG_LEVEL:                  INFO
      WATCH_NAMESPACE:            fi601-mcx-percona
      POD_NAME:                   psmdb-operator-6f995c5687-8ztzh (v1:metadata.name)
      OPERATOR_NAME:              percona-server-mongodb-operator
      RESYNC_PERIOD:              5s
      DISABLE_TELEMETRY:          true
      MAX_CONCURRENT_RECONCILES:  1
      PATH:                       /usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-597kn (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True
  Initialized                 True
  Ready                       False
  ContainersReady             False
  PodScheduled                True
Volumes:
  kube-api-access-597kn:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    Optional:                false
    DownwardAPI:             true
    ConfigMapName:           openshift-service-ca.crt
    Optional:                false
QoS Class:                   Guaranteed
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists for 60s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 60s
Events:
  Type     Reason  Age                     From     Message
  ----     ------  ----                    ----     -------
  Warning  Failed  8m20s (x5348 over 20h)  kubelet  (combined from similar events): Error: container create failed: time="2026-01-20T07:24:06Z" level=error msg="runc create failed: unable to start container process: exec: \"percona-server-mongodb-operator\": executable file not found in $PATH"
  Normal   Pulled  3m14s (x5379 over 20h)  kubelet  Container image "ncd-harbor.fi-601.tre.nsn-rdnet.net/mcx/percona/percona-server-mongodb-operator:1.21.1" already present on machine

Manually running the a Pod with the same image and overriding the command with sleep infinity allows me to exec into it, and doing ls /usr/local/bin shows that it is in fact empty. I don’t know how it is possible that the binary goes missing like that.

4

It turns out that the image got corrupted somehow when it was uploaded into the internal registry. Updating it with the correct one fixed the issue