# Authentication and liveness/readiness issues on new installation

**URL:** https://forums.percona.com/t/authentication-and-liveness-readiness-issues-on-new-installation/37771
**Category:** Percona Operator for MongoDB
**Created:** [April 29, 2025, 8:31am UTC](https://forums.percona.com/t/authentication-and-liveness-readiness-issues-on-new-installation/37771 "2025-04-29T08:31:07Z")
**Posts on this page:** 13
**Page:** 1

<div class="post-metadata">

### Author: ![Kay\_Khan](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/kay_khan/32/17023_2.png) [@Kay\_Khan](https://forums.percona.com/u/Kay_Khan)
#### Post date: [April 29, 2025, 8:31am UTC](https://forums.percona.com/t/authentication-and-liveness-readiness-issues-on-new-installation/37771/1 "2025-04-29T08:31:07Z")

</div>

## Description:

Ive deployed a new psmdb operator + mongodb to a new test cluster running aws eks 1.32.

However it is in a state of error

```auto
NAME ENDPOINT STATUS AGE
psmdb-default-sharded acme-prod-psmdb-default-sharded-572b8d37b9e51ccf.elb.us-east-2.amazonaws.com error 16h

```

```auto
Name: psmdb-default-sharded
Namespace: mongodb
Labels: app.kubernetes.io/instance=psmdb-default-sharded
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=psmdb-default-sharded
              app.kubernetes.io/version=1.19.1
              argocd.argoproj.io/instance=psmdb-default-sharded
              helm.sh/chart=psmdb-db-1.19.1
Annotations: <none>
API Version: psmdb.percona.com/v1
Kind: PerconaServerMongoDB
Metadata:
  Creation Timestamp: 2025-04-28T16:17:53Z
  Finalizers:
    percona.com/delete-psmdb-pods-in-order
  Generation: 2
  Resource Version: 1462659
  UID: f84ea80c-9a9d-4e89-9129-f71b98f8850d
Spec:
  Backup:
    Enabled: true
    Image: percona/percona-backup-mongodb:2.8.0-multi
    Pitr:
      Enabled: false
  Cr Version: 1.19.1
  Enable Volume Expansion: true
  Image: percona/percona-server-mongodb:8.0.4-2
  Image Pull Policy: Always
  Multi Cluster:
    Enabled: false
  Pause: false
  Pmm:
    Enabled: false
    Image: percona/pmm-client:2.44.0
    Server Host: monitoring-service
  Replsets:
    Affinity:
      Anti Affinity Topology Key: kubernetes.io/hostname
    Arbiter:
      Affinity:
        Anti Affinity Topology Key: kubernetes.io/hostname
      Enabled: false
      Size: 1
    Expose:
      Enabled: false
      Type: ClusterIP
    Name: rs0
    Node Selector:
      Karpenter - Node - Pool: mongodb-sharded
    Nonvoting:
      Affinity:
        Anti Affinity Topology Key: kubernetes.io/hostname
      Enabled: false
      Pod Disruption Budget:
        Max Unavailable: 1
      Resources:
        Limits:
          Cpu: 300m
          Memory: 0.5G
        Requests:
          Cpu: 300m
          Memory: 0.5G
      Size: 3
      Volume Spec:
        Persistent Volume Claim:
          Resources:
            Requests:
              Storage: 3Gi
    Pod Disruption Budget:
      Max Unavailable: 1
    Resources:
      Limits:
        Cpu: 2
        Memory: 4Gi
      Requests:
        Cpu: 300m
        Memory: 500M
    Size: 3
    Tolerations:
      Effect: NoSchedule
      Key: karpenter/mongodb-sharded
      Operator: Exists
    Volume Spec:
      Persistent Volume Claim:
        Resources:
          Requests:
            Storage: 100Gi
        Storage Class Name: mongodb
    Affinity:
    Expose:
      Enabled: false
    Name: rs1
    Node Selector:
      Karpenter - Node - Pool: mongodb-sharded
    Resources:
      Limits:
        Cpu: 2
        Memory: 4Gi
      Requests:
        Cpu: 300m
        Memory: 500M
    Size: 3
    Tolerations:
      Effect: NoSchedule
      Key: karpenter/mongodb-sharded
      Operator: Exists
    Volume Spec:
      Persistent Volume Claim:
        Resources:
          Requests:
            Storage: 100Gi
        Storage Class Name: mongodb
    Affinity:
    Expose:
      Enabled: false
    Name: rs2
    Node Selector:
      Karpenter - Node - Pool: mongodb-sharded
    Resources:
      Limits:
        Cpu: 2
        Memory: 4Gi
      Requests:
        Cpu: 300m
        Memory: 500M
    Size: 3
    Tolerations:
      Effect: NoSchedule
      Key: karpenter/mongodb-sharded
      Operator: Exists
    Volume Spec:
      Persistent Volume Claim:
        Resources:
          Requests:
            Storage: 100Gi
        Storage Class Name: mongodb
  Secrets:
    Users: psmdb-default-sharded-secrets
  Sharding:
    Balancer:
      Enabled: true
    Configsvr Repl Set:
      Affinity:
        Anti Affinity Topology Key: kubernetes.io/hostname
      Expose:
        Enabled: false
        Type: ClusterIP
      Node Selector:
        Karpenter - Node - Pool: mongodb-sharded
      Pod Disruption Budget:
        Max Unavailable: 1
      Resources:
        Limits:
          Cpu: 300m
          Memory: 0.5G
        Requests:
          Cpu: 300m
          Memory: 0.5G
      Size: 3
      Tolerations:
        Effect: NoSchedule
        Key: karpenter/mongodb-sharded
        Operator: Exists
      Volume Spec:
        Persistent Volume Claim:
          Resources:
            Requests:
              Storage: 3Gi
    Enabled: true
    Mongos:
      Affinity:
        Anti Affinity Topology Key: kubernetes.io/hostname
      Expose:
        Annotations:
          service.beta.kubernetes.io/aws-load-balancer-ip-address-type: ipv4
          service.beta.kubernetes.io/aws-load-balancer-name: acme-prod-psmdb-default-sharded
          service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: instance
          service.beta.kubernetes.io/aws-load-balancer-scheme: internal
        Type: LoadBalancer
      Node Selector:
        Karpenter - Node - Pool: mongodb-sharded
      Pod Disruption Budget:
        Max Unavailable: 1
      Resources:
        Limits:
          Cpu: 300m
          Memory: 0.5G
        Requests:
          Cpu: 300m
          Memory: 0.5G
      Size: 3
      Tolerations:
        Effect: NoSchedule
        Key: karpenter/mongodb-sharded
        Operator: Exists
  Unmanaged: false
  Unsafe Flags:
    Backup If Unhealthy: false
    Mongos Size: false
    Replset Size: false
    Termination Grace Period: false
    Tls: false
  Update Strategy: SmartUpdate
  Upgrade Options:
    Apply: disabled
    Schedule: 0 2 * * *
    Set FCV: false
    Version Service Endpoint: https://check.percona.com
Status:
  Conditions:
    Last Transition Time: 2025-04-28T16:17:53Z
    Status: True
    Type: sharding
    Last Transition Time: 2025-04-28T16:17:57Z
    Status: True
    Type: initializing
    Last Transition Time: 2025-04-28T16:22:12Z
    Message: create system users: failed to get mongo client: ping mongo: connection() error occurred during connection handshake: context canceled
    Reason: ErrorReconcile
    Status: True
    Type: error
    Last Transition Time: 2025-04-28T16:34:43Z
    Message: rs2: ready
    Reason: RSReady
    Status: True
    Type: ready
  Host: acme-prod-psmdb-default-sharded-572b8d37b9e51ccf.elb.us-east-2.amazonaws.com
  Message: Error: dial: ping mongo: connection() error occurred during connection handshake: auth error: sasl conversation error: unable to authenticate using mechanism "SCRAM-SHA-1": (AuthenticationFailed) Authentication failed.
  Mongos:
    Ready: 3
    Size: 3
    Status: ready
  Observed Generation: 2
  Ready: 14
  Replsets:
    Cfg:
      Initialized: true
      Members:
        psmdb-default-sharded-cfg-0:
          Name: psmdb-default-sharded-cfg-0.psmdb-default-sharded-cfg.mongodb.svc.cluster.local:27017
          State: 2
          State Str: SECONDARY
        psmdb-default-sharded-cfg-1:
          Name: psmdb-default-sharded-cfg-1.psmdb-default-sharded-cfg.mongodb.svc.cluster.local:27017
          State: 2
          State Str: SECONDARY
        psmdb-default-sharded-cfg-2:
          Name: psmdb-default-sharded-cfg-2.psmdb-default-sharded-cfg.mongodb.svc.cluster.local:27017
          State: 1
          State Str: PRIMARY
      Ready: 3
      Size: 3
      Status: ready
    rs0:
      added_as_shard: true
      Initialized: true
      Members:
        psmdb-default-sharded-rs0-0:
          Name: psmdb-default-sharded-rs0-0.psmdb-default-sharded-rs0.mongodb.svc.cluster.local:27017
          State: 2
          State Str: SECONDARY
        psmdb-default-sharded-rs0-1:
          Name: psmdb-default-sharded-rs0-1.psmdb-default-sharded-rs0.mongodb.svc.cluster.local:27017
          State: 2
          State Str: SECONDARY
        psmdb-default-sharded-rs0-2:
          Name: psmdb-default-sharded-rs0-2.psmdb-default-sharded-rs0.mongodb.svc.cluster.local:27017
          State: 1
          State Str: PRIMARY
      Ready: 3
      Size: 3
      Status: ready
    rs1:
      added_as_shard: true
      Initialized: true
      Members:
        psmdb-default-sharded-rs1-0:
          Name: psmdb-default-sharded-rs1-0.psmdb-default-sharded-rs1.mongodb.svc.cluster.local:27017
          State: 2
          State Str: SECONDARY
        psmdb-default-sharded-rs1-1:
          Name: psmdb-default-sharded-rs1-1.psmdb-default-sharded-rs1.mongodb.svc.cluster.local:27017
          State: 1
          State Str: PRIMARY
        psmdb-default-sharded-rs1-2:
          Name: psmdb-default-sharded-rs1-2.psmdb-default-sharded-rs1.mongodb.svc.cluster.local:27017
          State: 2
          State Str: SECONDARY
      Ready: 3
      Size: 3
      Status: ready
    rs2:
      added_as_shard: true
      Initialized: true
      Ready: 2
      Size: 3
      Status: initializing
  Size: 15
  State: error
Events: <none>

```

```auto
kubectl get all -n mongodb
NAME READY STATUS RESTARTS AGE
pod/psmdb-default-sharded-cfg-0 2/2 Running 0 12m
pod/psmdb-default-sharded-cfg-1 2/2 Running 0 10m
pod/psmdb-default-sharded-cfg-2 2/2 Running 0 9m2s
pod/psmdb-default-sharded-mongos-0 1/1 Running 0 8m59s
pod/psmdb-default-sharded-mongos-1 1/1 Running 0 12m
pod/psmdb-default-sharded-mongos-2 1/1 Running 0 10m
pod/psmdb-default-sharded-rs0-0 2/2 Running 0 12m
pod/psmdb-default-sharded-rs0-1 2/2 Running 0 10m
pod/psmdb-default-sharded-rs0-2 2/2 Running 0 9m2s
pod/psmdb-default-sharded-rs1-0 2/2 Running 0 12m
pod/psmdb-default-sharded-rs1-1 2/2 Running 0 10m
pod/psmdb-default-sharded-rs1-2 2/2 Running 0 9m2s
pod/psmdb-default-sharded-rs2-0 2/2 Running 0 12m
pod/psmdb-default-sharded-rs2-1 2/2 Running 0 10m
pod/psmdb-default-sharded-rs2-2 1/2 Running 3 (2s ago) 9m17s
pod/psmdb-operator-b6b76bbdc-g746t 1/1 Running 0 16h

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/psmdb-default-sharded-cfg ClusterIP None <none> 27017/TCP 16h
service/psmdb-default-sharded-metrics ClusterIP 172.20.93.242 <none> 9216/TCP 16h
service/psmdb-default-sharded-mongos LoadBalancer 172.20.95.86 acme-prod-psmdb-default-sharded-572b8d37b9e51ccf.elb.us-east-2.amazonaws.com 27017:31270/TCP 16h
service/psmdb-default-sharded-rs0 ClusterIP None <none> 27017/TCP 16h
service/psmdb-default-sharded-rs1 ClusterIP None <none> 27017/TCP 16h
service/psmdb-default-sharded-rs2 ClusterIP None <none> 27017/TCP 16h

NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/psmdb-operator 1/1 1 1 16h

NAME DESIRED CURRENT READY AGE
replicaset.apps/psmdb-operator-b6b76bbdc 1 1 1 16h

NAME READY AGE
statefulset.apps/psmdb-default-sharded-cfg 3/3 16h
statefulset.apps/psmdb-default-sharded-mongos 3/3 12h
statefulset.apps/psmdb-default-sharded-rs0 3/3 16h
statefulset.apps/psmdb-default-sharded-rs1 3/3 16h
statefulset.apps/psmdb-default-sharded-rs2 2/3 16h

```

[Detailed description of the issue or question]

## Steps to Reproduce:

```auto
fializers:
  - delete-psmdb-pods-in-order
nameOverride: ${db_name}
upgradeOptions:
  apply: disabled
image:
  repository: percona/percona-server-mongodb
  tag: 8.0.4-2
enableVolumeExpansion: true
replsets:
  rs0:
    name: rs0
    size: 3
    tolerations:
      - key: "karpenter/mongodb-sharded"
        operator: "Exists"
        effect: "NoSchedule"
    nodeSelector:
      karpenter-node-pool: mongodb-sharded
    affinity:
      podAntiAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchLabels:
                app.kubernetes.io/replset: rs0
            topologyKey: "kubernetes.io/hostname"
    resources: 
      limits:
        cpu: 2
        memory: 4Gi
      requests:
        cpu: 300m
        memory: 500M
    expose:
      enabled: false
    volumeSpec:
      pvc:
        storageClassName: mongodb
        resources:
          requests:
            storage: 100Gi
  rs1:
    name: rs1
    size: 3
    tolerations:
      - key: "karpenter/mongodb-sharded"
        operator: "Exists"
        effect: "NoSchedule"
    nodeSelector:
      karpenter-node-pool: mongodb-sharded
    affinity:
      podAntiAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchLabels:
                app.kubernetes.io/replset: rs0
            topologyKey: "kubernetes.io/hostname"
    resources: 
      limits:
        cpu: 2
        memory: 4Gi
      requests:
        cpu: 300m
        memory: 500M
    expose:
      enabled: false
    volumeSpec:
      pvc:
        storageClassName: mongodb
        resources:
          requests:
            storage: 100Gi
  rs2:
    name: rs2
    size: 3
    tolerations:
      - key: "karpenter/mongodb-sharded"
        operator: "Exists"
        effect: "NoSchedule"
    nodeSelector:
      karpenter-node-pool: mongodb-sharded
    affinity:
      podAntiAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchLabels:
                app.kubernetes.io/replset: rs0
            topologyKey: "kubernetes.io/hostname"
    resources: 
      limits:
        cpu: 2
        memory: 4Gi
      requests:
        cpu: 300m
        memory: 500M
    expose:
      enabled: false
    volumeSpec:
      pvc:
        storageClassName: mongodb
        resources:
          requests:
            storage: 100Gi
backup:
  enabled: true
sharding:
  enabled: true
  balancer:
    enabled: true
  configrs:
    size: 3
    tolerations:
      - key: "karpenter/mongodb-sharded"
        operator: "Exists"
        effect: "NoSchedule"
    nodeSelector:
      karpenter-node-pool: mongodb-sharded
    affinity:
      podAntiAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchLabels:
                app.kubernetes.io/component: cfg
  mongos:
    size: 3
    expose:
      enabled: true
      type: LoadBalancer
      annotations:
        service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: instance
        service.beta.kubernetes.io/aws-load-balancer-scheme: internal
        service.beta.kubernetes.io/aws-load-balancer-ip-address-type: ipv4
        service.beta.kubernetes.io/aws-load-balancer-name: acme-prod-psmdb-default-sharded
    tolerations:
      - key: "karpenter/mongodb-sharded"
        operator: "Exists"
        effect: "NoSchedule"
    nodeSelector:
      karpenter-node-pool: mongodb-sharded
    affinity:
      podAntiAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchLabels:
                app.kubernetes.io/component: mongos

```

## Version:

```auto
 helm ls -n mongodb
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
psmdb-operator mongodb 1 2025-04-28 17:17:01.915939422 +0100 BST deployed psmdb-operator-1.19.1 1.19.1

```

## Logs:

[If applicable, include any relevant log files or error messages]

## Expected Result:

[What the user expected to see or happen before the issue occurred]

## Actual Result:

[What actually happened when the user encountered the issue]

## Additional Information:

The issue only seems to be affecting a particular sharded set rs2,

- i’ve beefed up the size of the karpenter node incase it was a resoure issue.
- Ive done a restart of the rs2 , even killed the pods .

```auto
> 
> 
> 2025-04-29T08:31:26.864Z ERROR Failed to perform check {"error": "member failed Kubernetes liveness check: connection error: filed to dial mongo: ping mongo: connection() error occurred during connection handshake: auth error: sasl conversation error: unable to authenticate using mechanism \"SCRAM-SHA-1\": (AuthenticationFailed) Authentication failed.", "errorVerbose": "connection() error occurred during connection handshake: auth error: sasl conversation error: unable to authenticate using mechanism \"SCRAM-SHA-1\": (AuthenticationFailed) Authentication failed.\nping mongo\ngithub.com/percona/percona-server-mongodb-operator/pkg/psmdb/mongo.Dial\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/psmdb/mongo/mongo.go:123\ngithub.com/percona/percona-server-mongodb-operator/cmd/mongodb-healthcheck/db.Dial\n\t/go/src/github.com/percona/percona-server-mongodb-operator/cmd/mongodb-healthcheck/db/db.go:48\ngithub.com/percona/percona-server-mongodb-operator/cmd/mongodb-healthcheck/healthcheck.HealthCheckMongodLiveness\n\t/go/src/github.com/percona/percona-server-mongodb-operator/cmd/mongodb-healthcheck/healthcheck/health.go:62\ngithub.com/percona/percona-server-mongodb-operator/cmd/mongodb-healthcheck/tool.(*App).Run\n\t/go/src/github.com/percona/percona-server-mongodb-operator/cmd/mongodb-healthcheck/tool/tool.go:95\nmain.main\n\t/go/src/github.com/percona/percona-server-mongodb-operator/cmd/mongodb-healthcheck/main.go:67\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:272\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_arm64.s:1223\nfiled to dial mongo\ngithub.com/percona/percona-server-mongodb-operator/cmd/mongodb-healthcheck/db.Dial\n\t/go/src/github.com/percona/percona-server-mongodb-operator/cmd/mongodb-healthcheck/db/db.go:50\ngithub.com/percona/percona-server-mongodb-operator/cmd/mongodb-healthcheck/healthcheck.HealthCheckMongodLiveness\n\t/go/src/github.com/percona/percona-server-mongodb-operator/cmd/mongodb-healthcheck/healthcheck/health.go:62\ngithub.com/percona/percona-server-mongodb-operator/cmd/mongodb-healthcheck/tool.(*App).Run\n\t/go/src/github.com/percona/percona-server-mongodb-operator/cmd/mongodb-healthcheck/tool/tool.go:95\nmain.main\n\t/go/src/github.com/percona/percona-server-mongodb-operator/cmd/mongodb-healthcheck/main.go:67\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:272\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_arm64.s:1223\nconnection error\ngithub.com/percona/percona-server-mongodb-operator/cmd/mongodb-healthcheck/healthcheck.HealthCheckMongodLiveness\n\t/go/src/github.com/percona/percona-server-mongodb-operator/cmd/mongodb-healthcheck/healthcheck/health.go:64\ngithub.com/percona/percona-server-mongodb-operator/cmd/mongodb-healthcheck/tool.(*App).Run\n\t/go/src/github.com/percona/percona-server-mongodb-operator/cmd/mongodb-healthcheck/tool/tool.go:95\nmain.main\n\t/go/src/github.com/percona/percona-server-mongodb-operator/cmd/mongodb-healthcheck/main.go:67\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:272\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_arm64.s:1223\nmember failed Kubernetes liveness check"}
> main.main
> /go/src/github.com/percona/percona-server-mongodb-operator/cmd/mongodb-healthcheck/main.go:68
> runtime.main
> /usr/local/go/src/runtime/proc.go:272
```

---

<div class="post-metadata">

### Author: ![Slava\_Sarzhan](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/slava_sarzhan/32/3196_2.png) [@Slava\_Sarzhan](https://forums.percona.com/u/Slava_Sarzhan)
#### Post date: [April 29, 2025, 11:30am UTC](https://forums.percona.com/t/authentication-and-liveness-readiness-issues-on-new-installation/37771/2 "2025-04-29T11:30:08Z")

</div>

> [@Kay\_Khan](#):
>
> `2 1/2 `

Hi, it’s strange. I can’t reproduce this issue. Try deleting this pod with PVC to force reinitit it from scratch.

---

<div class="post-metadata">

### Author: ![Kay\_Khan](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/kay_khan/32/17023_2.png) [@Kay\_Khan](https://forums.percona.com/u/Kay_Khan)
#### Post date: [April 29, 2025, 1:07pm UTC](https://forums.percona.com/t/authentication-and-liveness-readiness-issues-on-new-installation/37771/3 "2025-04-29T13:07:26Z")

</div>

thank you for the suggestion, i tried that but unfortunately still have the same issue.

crashing a reboots shortly after

 ![image](https://us1.discourse-cdn.com/flex019/uploads/percona1/original/3X/e/2/e21db582c68bfe0e31736ac470584009d26447d9.png)

I noticed a few strange logs in the pod that maybe the cause issues?

```auto
{"t":{"$date":"2025-04-29T13:05:23.463+00:00"},"s":"I", "c":"ACCESS", "id":5286307, "svc":"S", "ctx":"conn71","msg":"Failed to authenticate","attr":{"client":"10.2.164.181:55230","isSpeculative":true,"isClusterMember":false,"mechanism":"SCRAM-SHA-256","user":"backup","db":"admin","error":"UserNotFound: Could not find user \"backup\" for db \"admin\"","result":11,"metrics":{"conversation_duration":{"micros":162,"summary":{"0":{"step":1,"step_total":2,"duration_micros":140}}}},"extraInfo":{}}}
{"t":{"$date":"2025-04-29T13:05:23.464+00:00"},"s":"I", "c":"ACCESS", "id":6788604, "svc":"S", "ctx":"conn71","msg":"Auth metrics report","attr":{"metric":"acquireUser","micros":0}}
{"t":{"$date":"2025-04-29T13:05:23.464+00:00"},"s":"I", "c":"ACCESS", "id":5286307, "svc":"S", "ctx":"conn71","msg":"Failed to authenticate","attr":{"client":"10.2.164.181:55230","isSpeculative":false,"isClusterMember":false,"mechanism":"SCRAM-SHA-1","user":"backup","db":"admin","error":"UserNotFound: Could not find user \"backup\" for db \"admin\"","result":11,"metrics":{"conversation_duration":{"micros":152,"summary":{"0":{"step":1,"step_total":2,"duration_micros":134}}}},"extraInfo":{}}}
{"t":{"$date":"2025-04-29T13:05:23.464+00:00"},"s":"I", "c":"-", "id":20883, "svc":"S", "ctx":"conn70","msg":"Interrupted operation as its client disconnected","attr":{"opId":117761}}
{"t":{"$date":"2025-04-29T13:05:23.465+00:00"},"s":"I", "c":"NETWORK", "id":6723804, "svc":"S", "ctx":"conn72","msg":"Ingress TLS handshake complete","attr":{"durationMillis":5}}
{"t":{"$date":"2025-04-29T13:05:23.465+00:00"},"s":"W", "c":"NETWORK", "id":23236, "svc":"S", "ctx":"conn72","msg":"Client connecting with server's own TLS certificate"}
{"t":{"$date":"2025-04-29T13:05:23.488+00:00"},"s":"I", "c":"-", "id":4939300, "svc":"S", "ctx":"monitoring-keys-for-HMAC","msg":"Failed to refresh key cache","attr":{"error":"KeyNotFound: No keys found after refresh","nextWakeupMillis":3600}}
{"t":{"$date":"2025-04-29T13:05:27.088+00:00"},"s":"I", "c":"-", "id":4939300, "svc":"S", "ctx":"monitoring-keys-for-HMAC","msg":"Failed to refresh key cache","attr":{"error":"KeyNotFound: No keys found after refresh","nextWakeupMillis":3800}}

```

---

<div class="post-metadata">

### Author: ![Kay\_Khan](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/kay_khan/32/17023_2.png) [@Kay\_Khan](https://forums.percona.com/u/Kay_Khan)
#### Post date: [April 29, 2025, 2:07pm UTC](https://forums.percona.com/t/authentication-and-liveness-readiness-issues-on-new-installation/37771/4 "2025-04-29T14:07:32Z")

</div>

if i remove rs2 im able to get a healthy and ready mongodb cluster, but for whatever reason when i readd rs2 the cluster is degraded and 1 or more rs2 pods are crashing and rebooting

---

<div class="post-metadata">

### Author: ![Slava\_Sarzhan](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/slava_sarzhan/32/3196_2.png) [@Slava\_Sarzhan](https://forums.percona.com/u/Slava_Sarzhan)
#### Post date: [April 29, 2025, 2:24pm UTC](https://forums.percona.com/t/authentication-and-liveness-readiness-issues-on-new-installation/37771/5 "2025-04-29T14:24:29Z")

</div>

Do you have data in your cluster?

---

<div class="post-metadata">

### Author: ![Kay\_Khan](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/kay_khan/32/17023_2.png) [@Kay\_Khan](https://forums.percona.com/u/Kay_Khan)
#### Post date: [April 29, 2025, 3:29pm UTC](https://forums.percona.com/t/authentication-and-liveness-readiness-issues-on-new-installation/37771/7 "2025-04-29T15:29:53Z")

</div>

There is no data in the cluster, its all fresh/new

---

<div class="post-metadata">

### Author: ![Slava\_Sarzhan](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/slava_sarzhan/32/3196_2.png) [@Slava\_Sarzhan](https://forums.percona.com/u/Slava_Sarzhan)
#### Post date: [April 29, 2025, 5:38pm UTC](https://forums.percona.com/t/authentication-and-liveness-readiness-issues-on-new-installation/37771/8 "2025-04-29T17:38:30Z")

</div>

> [@Kay\_Khan](#):
>
> `Auth metrics report`

Maybe the node that is used for rs2-2 has some connectivity problems..

---

<div class="post-metadata">

### Author: ![Kay\_Khan](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/kay_khan/32/17023_2.png) [@Kay\_Khan](https://forums.percona.com/u/Kay_Khan)
#### Post date: [May 1, 2025, 8:45am UTC](https://forums.percona.com/t/authentication-and-liveness-readiness-issues-on-new-installation/37771/9 "2025-05-01T08:45:48Z")

</div>

I think I might have figured it out.

I was previously using the image with `tag: 8.0.4-1-multi` to support r7g EC2 instance types (Graviton/ARM). After fully deleting the cluster—along with any leftover volumes—and updating the image to `tag: 8.0.4-2` (which no longer has a `-multi` variant), everything started working again on the same EC2 instance types.

[https://hub.docker.com/r/percona/percona-server-mongodb/tags](https://hub.docker.com/r/percona/percona-server-mongodb/tags)

---

<div class="post-metadata">

### Author: ![Slava\_Sarzhan](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/slava_sarzhan/32/3196_2.png) [@Slava\_Sarzhan](https://forums.percona.com/u/Slava_Sarzhan)
#### Post date: [May 1, 2025, 10:41am UTC](https://forums.percona.com/t/authentication-and-liveness-readiness-issues-on-new-installation/37771/10 "2025-05-01T10:41:08Z")

</div>

@Kay_Khan good news. Thank you for the update.

---

<div class="post-metadata">

### Author: ![Kay\_Khan](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/kay_khan/32/17023_2.png) [@Kay\_Khan](https://forums.percona.com/u/Kay_Khan)
#### Post date: [May 2, 2025, 9:51am UTC](https://forums.percona.com/t/authentication-and-liveness-readiness-issues-on-new-installation/37771/11 "2025-05-02T09:51:45Z")

</div>

Yes it is good news, although im unsrue why i dont need to specify the -multi variant of the image given that im running on r7g large which is arm based ? do you have any ideas?

---

<div class="post-metadata">

### Author: ![Slava\_Sarzhan](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/slava_sarzhan/32/3196_2.png) [@Slava\_Sarzhan](https://forums.percona.com/u/Slava_Sarzhan)
#### Post date: [May 2, 2025, 10:31am UTC](https://forums.percona.com/t/authentication-and-liveness-readiness-issues-on-new-installation/37771/12 "2025-05-02T10:31:29Z")

</div>

> [@Kay\_Khan](#):
>
> 8.0.4-1

As you can see [https://hub.docker.com/r/percona/percona-server-mongodb/tags?name=8.0.4-1](https://hub.docker.com/r/percona/percona-server-mongodb/tags?name=8.0.4-1)  
[8.0.4-1](https://hub.docker.com/layers/percona/percona-server-mongodb/8.0.4-1/images/sha256-873b201ce3d66d97b1225c26db392c5043a73cc19ee8db6f2dc1b8efd4783bcf) baced on linux/amd64. Maybe one of your nodes based on and…

---

<div class="post-metadata">

### Author: ![Kay\_Khan](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/kay_khan/32/17023_2.png) [@Kay\_Khan](https://forums.percona.com/u/Kay_Khan)
#### Post date: [May 2, 2025, 10:02pm UTC](https://forums.percona.com/t/authentication-and-liveness-readiness-issues-on-new-installation/37771/13 "2025-05-02T22:02:39Z")

</div>

> [@Kay\_Khan](#):
>
> r7g

all of our nodes are r7g arm based

---

<div class="post-metadata">

### Author: ![Sam\_Lin](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/sam_lin/32/20714_2.png) [@Sam\_Lin](https://forums.percona.com/u/Sam_Lin)
#### Post date: [May 19, 2025, 10:57pm UTC](https://forums.percona.com/t/authentication-and-liveness-readiness-issues-on-new-installation/37771/14 "2025-05-19T22:57:33Z")

</div>

I have similar issue as well. It works, if I delete all previous pvc but I think it should work without doing that. I use the same users secrets and encryption key.