Installation of PSMDB resource failed with TLS error

Installation of PSMDB fails with TLS error.
This is fresh installation and followed the below procedure:

  1. Deploy the operator using the following command:
$ kubectl apply --server-side -f https://raw.githubusercontent.com/percona/percona-server-mongodb-operator/v1.19.1/deploy/bundle.yaml
  1. Deploy MongoDB cluster with:
$ kubectl apply -f https://raw.githubusercontent.com/percona/percona-server-mongodb-operator/v1.19.1/deploy/cr-minimal.yaml

It failed with below error:

2025-03-20T12:01:39.861Z        INFO    createSSLByCertManager  updating cert-manager certificates      {"controller": "psmdb-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDB", "PerconaServerMongoDB": {"name":"cluster-3","namespace":"mongodb-test"}, "namespace": "mongodb-test", "name": "cluster-3", "reconcileID": "4aa5e6ed-25a6-4ff3-8daf-d11653e20c8c"}
2025-03-20T12:01:39.861Z        INFO    Creating old secrets    {"controller": "psmdb-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDB", "PerconaServerMongoDB": {"name":"cluster-3","namespace":"mongodb-test"}, "namespace": "mongodb-test", "name": "cluster-3", "reconcileID": "4aa5e6ed-25a6-4ff3-8daf-d11653e20c8c"}
2025-03-20T12:01:39.881Z        INFO    applying new certificates       {"controller": "psmdb-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDB", "PerconaServerMongoDB": {"name":"cluster-3","namespace":"mongodb-test"}, "namespace": "mongodb-test", "name": "cluster-3", "reconcileID": "4aa5e6ed-25a6-4ff3-8daf-d11653e20c8c"}
2025-03-20T12:01:40.943Z        ERROR   Reconciler error        {"controller": "psmdb-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDB", "PerconaServerMongoDB": {"name":"cluster-3","namespace":"mongodb-test"}, "namespace": "mongodb-test", "name": "cluster-3", "reconcileID": "4aa5e6ed-25a6-4ff3-8daf-d11653e20c8c", "error": "TLS secrets handler: \"create ssl by cert-manager: update cert mangager certs: failed to apply cert-manager certificates: failed to wait for ca cert: set controller reference: Object mongodb-test/cluster-3-ca-cert is already owned by another Certificate controller cluster-3-ca-cert\". Please create your TLS secret cluster-3-ssl manually or setup cert-manager correctly", "errorVerbose": "TLS secrets handler: \"create ssl by cert-manager: update cert mangager certs: failed to apply cert-manager certificates: failed to wait for ca cert: set controller reference: Object mongodb-test/cluster-3-ca-cert is already owned by another Certificate controller cluster-3-ca-cert\". Please create your TLS secret cluster-3-ssl manually or setup cert-manager correctly\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:389\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/controller/controller.go:116\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/controller/controller.go:303\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/controller/controller.go:263\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/controller/controller.go:224\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1700"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/controller/controller.go:316
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/controller/controller.go:263
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/controller/controller.go:224
2025-03-20T12:01:51.438Z        INFO    createSSLByCertManager  updating cert-manager certificates      {"controller": "psmdb-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDB", "PerconaServerMongoDB": {"name":"cluster-3","namespace":"mongodb-test"}, "namespace": "mongodb-test", "name": "cluster-3", "reconcileID": "4180d379-d2ba-4fee-a662-6f71b94693f5"}
2025-03-20T12:01:51.438Z        INFO    Creating old secrets    {"controller": "psmdb-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDB", "PerconaServerMongoDB": {"name":"cluster-3","namespace":"mongodb-test"}, "namespace": "mongodb-test", "name": "cluster-3", "reconcileID": "4180d379-d2ba-4fee-a662-6f71b94693f5"}
2025-03-20T12:01:51.449Z        INFO    applying new certificates       {"controller": "psmdb-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDB", "PerconaServerMongoDB": {"name":"cluster-3","namespace":"mongodb-test"}, "namespace": "mongodb-test", "name": "cluster-3", "reconcileID": "4180d379-d2ba-4fee-a662-6f71b94693f5"}
2025-03-20T12:01:52.503Z        ERROR   Reconciler error        {"controller": "psmdb-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDB", "PerconaServerMongoDB": {"name":"cluster-3","namespace":"mongodb-test"}, "namespace": "mongodb-test", "name": "cluster-3", "reconcileID": "4180d379-d2ba-4fee-a662-6f71b94693f5", "error": "TLS secrets handler: \"create ssl by cert-manager: update cert mangager certs: failed to apply cert-manager certificates: failed to wait for ca cert: set controller reference: Object mongodb-test/cluster-3-ca-cert is already owned by another Certificate controller cluster-3-ca-cert\". Please create your TLS secret cluster-3-ssl manually or setup cert-manager correctly", "errorVerbose": "TLS secrets handler: \"create ssl by cert-manager: update cert mangager certs: failed to apply cert-manager certificates: failed to wait for ca cert: set controller reference: Object mongodb-test/cluster-3-ca-cert is already owned by another Certificate controller cluster-3-ca-cert\". Please create your TLS secret cluster-3-ssl manually or setup cert-manager correctly\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:389\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/controller/controller.go:116\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/controller/controller.go:303\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/controller/controller.go:263\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/controller/controller.go:224\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1700"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/controller/controller.go:316
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/controller/controller.go:263
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/controller/controller.go:224


===================================================
kubectl describe psmdb cluster-3 -n mongodb-test
Name:         cluster-3
Namespace:    mongodb-test
Labels:       <none>
Annotations:  <none>
API Version:  psmdb.percona.com/v1
Kind:         PerconaServerMongoDB
Metadata:
  Creation Timestamp:  2025-03-20T12:01:22Z
  Finalizers:
    percona.com/delete-psmdb-pods-in-order
  Generation:        1
  Resource Version:  80855269
  UID:               16703d55-151e-4be1-a04a-c2df58b2ef2e
Spec:
  Backup:
    Enabled:  true
    Image:    percona/percona-backup-mongodb:2.8.0-multi
    Pitr:
      Compression Level:  6
      Compression Type:   gzip
      Enabled:            false
      Oplog Only:         false
  Cr Version:             1.19.1
  Image:                  percona/percona-server-mongodb:7.0.15-9-multi
  Image Pull Policy:      Always
  Pmm:
    Enabled:      false
    Image:        percona/pmm-client:2.44.0
    Server Host:  monitoring-service
  Replsets:
    Affinity:
      Anti Affinity Topology Key:  kubernetes.io/hostname
    Arbiter:
      Affinity:
        Anti Affinity Topology Key:  kubernetes.io/hostname
      Enabled:                       false
      Resources:
        Limits:
          Cpu:     300m
          Memory:  0.5G
        Requests:
          Cpu:     300m
          Memory:  0.5G
      Size:        1
    Expose:
      Enabled:  false
      Type:     ClusterIP
    Name:       rs0
    Nonvoting:
      Affinity:
        Anti Affinity Topology Key:  kubernetes.io/hostname
      Enabled:                       false
      Pod Disruption Budget:
        Max Unavailable:  1
      Resources:
        Limits:
          Cpu:     300m
          Memory:  0.5G
        Requests:
          Cpu:     300m
          Memory:  0.5G
      Size:        3
      Volume Spec:
        Persistent Volume Claim:
          Resources:
            Requests:
              Storage:  3Gi
    Pod Disruption Budget:
      Max Unavailable:  1
    Resources:
      Limits:
        Cpu:     300m
        Memory:  0.5G
      Requests:
        Cpu:     300m
        Memory:  0.5G
    Size:        3
    Volume Spec:
      Persistent Volume Claim:
        Resources:
          Requests:
            Storage:  3Gi
  Secrets:
    Encryption Key:  cluster-3-mongodb-encryption-key
    Users:           cluster-3-secrets
  Sharding:
    Configsvr Repl Set:
      Affinity:
        Anti Affinity Topology Key:  kubernetes.io/hostname
      Expose:
        Enabled:  false
        Type:     ClusterIP
      Pod Disruption Budget:
        Max Unavailable:  1
      Resources:
        Limits:
          Cpu:     300m
          Memory:  0.5G
        Requests:
          Cpu:     300m
          Memory:  0.5G
      Size:        3
      Volume Spec:
        Persistent Volume Claim:
          Resources:
            Requests:
              Storage:  3Gi
    Enabled:            true
    Mongos:
      Affinity:
        Anti Affinity Topology Key:  kubernetes.io/hostname
      Expose:
        Type:  ClusterIP
      Pod Disruption Budget:
        Max Unavailable:  1
      Resources:
        Limits:
          Cpu:     300m
          Memory:  0.5G
        Requests:
          Cpu:      300m
          Memory:   0.5G
      Size:         3
  Update Strategy:  SmartUpdate
  Upgrade Options:
    Apply:                     disabled
    Schedule:                  0 2 * * *
    Set FCV:                   false
    Version Service Endpoint:  https://check.percona.com
Status:
  Conditions:
    Last Transition Time:  2025-03-20T12:01:22Z
    Status:                True
    Type:                  sharding
    Last Transition Time:  2025-03-20T12:01:24Z
    Message:               TLS secrets handler: "create ssl by cert-manager: update cert mangager certs: failed to apply cert-manager certificates: failed to wait for ca cert: set controller reference: Object mongodb-test/cluster-3-ca-cert is already owned by another Certificate controller cluster-3-ca-cert". Please create your TLS secret cluster-3-ssl manually or setup cert-manager correctly
    Reason:                ErrorReconcile
    Status:                True
    Type:                  error
  Message:                 Error: TLS secrets handler: "create ssl by cert-manager: update cert mangager certs: failed to apply cert-manager certificates: failed to wait for ca cert: set controller reference: Object mongodb-test/cluster-3-ca-cert is already owned by another Certificate controller cluster-3-ca-cert". Please create your TLS secret cluster-3-ssl manually or setup cert-manager correctly
  Ready:                   0
  Size:                    0
  State:                   error
Events:                    <none>


================

I can see the secret & cert is there.

 kubectl get secret -n mongodb-test
NAME                       TYPE                DATA   AGE
cluster-3-ca-cert          kubernetes.io/tls   3      53s
cluster-3-ca-cert-old      Opaque              3      3m24s
cluster-3-secrets          Opaque              10     10m
internal-cluster-3-users   Opaque              20     10m

Hi @surajm, As I can see, you have a cert-manager. Am I right? How did you deploy it, and which version is used?

Hi @Slava_Sarzhan ,

Yes, I have cert-manager in my deployment. It was deployed using helm chart. Steps below for reference:

helm repo add cert-manager https://charts.jetstack.io
cat << EOF > $values_yaml
installCRDs: true
extraArgs:
  - --enable-certificate-owner-ref=true
EOF
helm upgrade --install --atomic \
        cert-manager cert-manager/cert-manager --version v1.14.5 \
        --values $values_yaml

Version used is: Quay

The certs for this MongoDB deployment were generated using the procedure followed from Percona documentation:

I am able to install it now using certs generated by following the Percona documentation.
But I see other problems here.

  1. MongoS service failed on Unmanaged cluster (Cluster-2) in MongoDB deployment over MultiCluster with ISTIO Service Mesh with Shard enable.

  2. ReplSet from Cluster-2 is showing as Unhealthy in Cluster-1 even though these are reachable when checked via openssl, curl command.

More details below:

I have requirement to deploy MongoDB with Sharding over multiple K8S clusters. K8S clusters are connected via ISTIO mesh. East-West traffic across clusters flow via East-West Gateway in Istio Mesh.

I deployed Percona Operator and MongoDB with Shard enable. Also, exposed the ReplSet, Config Server and MongoS. Generated the certs using the procedure mentioned in Percona documentation : Transport encryption (TLS/SSL) - Percona Operator for MongoDB

Also, added the external nodes from Cluster-2 in Cluster-1 deployment yaml.
Below is the sample deployment file for Cluster-1:

apiVersion: psmdb.percona.com/v1
kind: PerconaServerMongoDB
metadata:
  name: cluster-1
  finalizers:
    - percona.com/delete-psmdb-pods-in-order
spec:
  clusterServiceDNSMode: "ServiceMesh"
  crVersion: 1.19.1
  image: percona/percona-server-mongodb:7.0.15-9-multi
  imagePullPolicy: Always
  updateStrategy: SmartUpdate
  upgradeOptions:
    versionServiceEndpoint: https://check.percona.com
    apply: disabled
    schedule: "0 2 * * *"
    setFCV: false
  secrets:
    users: cluster-1-secrets
    encryptionKey: cluster-1-mongodb-encryption-key
    ssl: cluster-1-ssl
    sslInternal: cluster-1-ssl-internal
    sse: cluster-1-sse
  pmm:
    enabled: false
    image: percona/pmm-client:2.44.0
    serverHost: monitoring-service
  replsets:
  - name: rs0
    size: 3
    replsetOverrides:
      cluster-1-rs0-0:
        host: cluster-1-rs0-0.mongodb-test-c1.svc.cluster.local:27017
        priority: 3
        tags:
          key: cluster-1-value-0
      cluster-1-rs0-1:
        host: cluster-1-rs0-1.mongodb-test-c1.svc.cluster.local:27017
        tags:
          key: cluster-1-value-1
      cluster-1-rs0-2:
        host: cluster-1-rs0-2.mongodb-test-c1.svc.cluster.local:27017
        tags:
          key: cluster-1-value-2
    externalNodes:
    - host: cluster-2-rs0-0.mongodb-test-1.svc.cluster.local
      port: 15443
      votes: 0
      priority: 0
    - host: cluster-2-rs0-1.mongodb-test-1.svc.cluster.local
      port: 15443
      votes: 0
      priority: 0
    - host: cluster-2-rs0-2.mongodb-test-1.svc.cluster.local
      port: 15443
      votes: 0
      priority: 0
    affinity:
      antiAffinityTopologyKey: "kubernetes.io/hostname"
    podDisruptionBudget:
      maxUnavailable: 1
    expose:
      enabled: true
      type: ClusterIP
    resources:
      limits:
        cpu: "300m"
        memory: "0.5G"
      requests:
        cpu: "300m"
        memory: "0.5G"
    volumeSpec:
      persistentVolumeClaim:
        resources:
          requests:
            storage: 3Gi

    nonvoting:
      enabled: false
      size: 3
      affinity:
        antiAffinityTopologyKey: "kubernetes.io/hostname"
      podDisruptionBudget:
        maxUnavailable: 1
      resources:
        limits:
          cpu: "300m"
          memory: "0.5G"
        requests:
          cpu: "300m"
          memory: "0.5G"
      volumeSpec:
        persistentVolumeClaim:
          resources:
            requests:
              storage: 3Gi
    arbiter:
      enabled: false
      size: 1
      affinity:
        antiAffinityTopologyKey: "kubernetes.io/hostname"
      resources:
        limits:
          cpu: "300m"
          memory: "0.5G"
        requests:
          cpu: "300m"
          memory: "0.5G"

  sharding:
    enabled: true
    configsvrReplSet:
      expose:
        enabled: true
        type: ClusterIP
      size: 3
      externalNodes:
      - host: cluster-2-cfg-0.mongodb-test-1.svc.cluster.local
        port: 15443
        votes: 0
        priority: 0
      - host: cluster-2-cfg-1.mongodb-test-1.svc.cluster.local
        port: 15443
        votes: 0
        priority: 0
      - host: cluster-2-cfg-2.mongodb-test-1.svc.cluster.local
        port: 15443
        votes: 0
        priority: 0
      affinity:
        antiAffinityTopologyKey: "kubernetes.io/hostname"
      podDisruptionBudget:
        maxUnavailable: 1
      resources:
        limits:
          cpu: "300m"
          memory: "0.5G"
        requests:
          cpu: "300m"
          memory: "0.5G"
      volumeSpec:
        persistentVolumeClaim:
          resources:
            requests:
              storage: 3Gi

    mongos:
      expose:
        enabled: true
        type: ClusterIP
      size: 3
      affinity:
        antiAffinityTopologyKey: "kubernetes.io/hostname"
      podDisruptionBudget:
        maxUnavailable: 1
      resources:
        limits:
          cpu: "300m"
          memory: "0.5G"
        requests:
          cpu: "300m"
          memory: "0.5G"
      expose:
        type: ClusterIP

  backup:
    enabled: true
    image: percona/percona-backup-mongodb:2.8.0-multi
    pitr:
      enabled: false
      oplogOnly: false
      compressionType: gzip
      compressionLevel: 6

All pods ReplSet, Config Server and MongoS came up fine and able to connect to MongoS as well from this Cluster-1.

now deployed the Percona Operator and MongoDB in Cluster-2 as per the documentation. Used the same certs, keys from Cluster-1 into this Cluster-2.

Below is the sample deployment yaml for Cluster-2:

apiVersion: psmdb.percona.com/v1
kind: PerconaServerMongoDB
metadata:
  name: cluster-2
  finalizers:
    - percona.com/delete-psmdb-pods-in-order
spec:
  clusterServiceDNSMode: "ServiceMesh"
  unmanaged: true
  crVersion: 1.19.1
  image: percona/percona-server-mongodb:7.0.15-9-multi
  imagePullPolicy: Always
  updateStrategy: OnDelete
  upgradeOptions:
    versionServiceEndpoint: https://check.percona.com
    apply: disabled
    schedule: "0 2 * * *"
    setFCV: false
  secrets:
    users: cluster-2-secrets
    encryptionKey: cluster-2-mongodb-encryption-key
    ssl: cluster-2-ssl
    sslInternal: cluster-2-ssl-internal
    keyFile: cluster-2-mongodb-keyfile
    sse: cluster-2-sse
  pmm:
    enabled: false
    image: percona/pmm-client:2.44.0
    serverHost: monitoring-service
  replsets:
  - name: rs0
    size: 3
    replsetOverrides:
      cluster-2-rs0-0:
        host: cluster-2-rs0-0.mongodb-test-1.svc.cluster.local:27017
        priority: 3
        tags:
          key: cluster-2-value-0
      cluster-2-rs0-1:
        host: cluster-2-rs0-1.mongodb-test-1.svc.cluster.local:27017
        tags:
          key: cluster-2-value-1
      cluster-2-rs0-2:
        host: cluster-2-rs0-2.mongodb-test-1.svc.cluster.local:27017
        tags:
          key: cluster-2-value-2
    affinity:
      antiAffinityTopologyKey: "kubernetes.io/hostname"
    podDisruptionBudget:
      maxUnavailable: 1
    expose:
      enabled: true
      type: ClusterIP
    resources:
      limits:
        cpu: "300m"
        memory: "0.5G"
      requests:
        cpu: "300m"
        memory: "0.5G"
    volumeSpec:
      persistentVolumeClaim:
        resources:
          requests:
            storage: 3Gi

    nonvoting:
      enabled: false
      size: 3
      affinity:
        antiAffinityTopologyKey: "kubernetes.io/hostname"
      podDisruptionBudget:
        maxUnavailable: 1
      resources:
        limits:
          cpu: "300m"
          memory: "0.5G"
        requests:
          cpu: "300m"
          memory: "0.5G"
      volumeSpec:
        persistentVolumeClaim:
          resources:
            requests:
              storage: 3Gi
    arbiter:
      enabled: false
      size: 1
      affinity:
        antiAffinityTopologyKey: "kubernetes.io/hostname"
      resources:
        limits:
          cpu: "300m"
          memory: "0.5G"
        requests:
          cpu: "300m"
          memory: "0.5G"

  sharding:
    enabled: true
    configsvrReplSet:
      expose:
        enabled: true
        type: ClusterIP
      size: 3
      affinity:
        antiAffinityTopologyKey: "kubernetes.io/hostname"
      podDisruptionBudget:
        maxUnavailable: 1
      resources:
        limits:
          cpu: "300m"
          memory: "0.5G"
        requests:
          cpu: "300m"
          memory: "0.5G"
      volumeSpec:
        persistentVolumeClaim:
          resources:
            requests:
              storage: 3Gi

    mongos:
      expose:
        enabled: true
        type: ClusterIP
      size: 3
      affinity:
        antiAffinityTopologyKey: "kubernetes.io/hostname"
      podDisruptionBudget:
        maxUnavailable: 1
      resources:
        limits:
          cpu: "300m"
          memory: "0.5G"
        requests:
          cpu: "300m"
          memory: "0.5G"
      expose:
        type: ClusterIP

  backup:
    enabled: false
    image: percona/percona-backup-mongodb:2.8.0-multi
    pitr:
      enabled: false
      oplogOnly: false
      compressionType: gzip
      compressionLevel: 6

Below errors seen in MongoS pod in Cluster-2:

{"t":{"$date":"2025-03-25T09:53:28.791+00:00"},"s":"I",  "c":"NETWORK",  "id":4333208, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"RSM host selection timeout","attr":{"replicaSet":"cfg","error":"FailedToSatisfyReadPreference: Could not find host matching read preference { mode: \"nearest\" } for set cfg"}}
{"t":{"$date":"2025-03-25T09:53:28.791+00:00"},"s":"W",  "c":"SHARDING", "id":23834,   "ctx":"mongosMain","msg":"Error loading global settings from config server. Sleeping for 2 seconds and retrying","attr":{"error":{"code":133,"codeName":"FailedToSatisfyReadPreference","errmsg":"Error loading clusterID :: caused by :: Could not find host matching read preference { mode: \"nearest\" } for set cfg"}}}
{"t":{"$date":"2025-03-25T09:53:33.487+00:00"},"s":"I",  "c":"NETWORK",  "id":4333208, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"RSM host selection timeout","attr":{"replicaSet":"cfg","error":"FailedToSatisfyReadPreference: Could not find host matching read preference { mode: \"nearest\" } for set cfg"}}
{"t":{"$date":"2025-03-25T09:53:33.487+00:00"},"s":"I",  "c":"-",        "id":4939300, "ctx":"monitoring-keys-for-HMAC","msg":"Failed to refresh key cache","attr":{"error":"FailedToSatisfyReadPreference: Could not find host matching read preference { mode: \"nearest\" } for set cfg","nextWakeupMillis":50000}}
{"t":{"$date":"2025-03-25T09:53:41.792+00:00"},"s":"I",  "c":"NETWORK",  "id":4333208, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"RSM host selection timeout","attr":{"replicaSet":"cfg","error":"FailedToSatisfyReadPreference: Could not find host matching read preference { mode: \"primary\" } for set cfg"}}
{"t":{"$date":"2025-03-25T09:53:41.792+00:00"},"s":"I",  "c":"SHARDING", "id":6973904, "ctx":"QueryAnalysisConfigurationsRefresher","msg":"Failed to refresh query analysis configurations, will try again at the next refresh interval","attr":{"error":"FailedToSatisfyReadPreference: Could not find host matching read preference { mode: \"primary\" } for set cfg"}}
{"t":{"$date":"2025-03-25T09:53:45.791+00:00"},"s":"I",  "c":"NETWORK",  "id":4333208, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"RSM host selection timeout","attr":{"replicaSet":"cfg","error":"FailedToSatisfyReadPreference: Could not find host matching read preference { mode: \"nearest\" } for set cfg"}}
{"t":{"$date":"2025-03-25T09:53:45.791+00:00"},"s":"W",  "c":"SHARDING", "id":23834,   "ctx":"mongosMain","msg":"Error loading global settings from config server. Sleeping for 2 seconds and retrying","attr":{"error":{"code":133,"codeName":"FailedToSatisfyReadPreference","errmsg":"Error loading clusterID :: caused by :: Could not find host matching read preference { mode: \"nearest\" } for set cfg"}}}
{"t":{"$date":"2025-03-25T09:53:56.693+00:00"},"s":"I",  "c":"NETWORK",  "id":4333208, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"RSM host selection timeout","attr":{"replicaSet":"cfg","error":"FailedToSatisfyReadPreference: Could not find host matching read preference { mode: \"nearest\" } for set cfg"}}
{"t":{"$date":"2025-03-25T09:53:56.693+00:00"},"s":"I",  "c":"SHARDING", "id":22727,   "ctx":"ShardRegistryUpdater","msg":"Error running periodic reload of shard registry","attr":{"error":"FailedToSatisfyReadPreference: Could not find host matching read preference { mode: \"nearest\" } for set cfg","shardRegistryReloadIntervalSeconds":30}}
{"t":{"$date":"2025-03-25T09:53:56.792+00:00"},"s":"I",  "c":"NETWORK",  "id":4333208, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"RSM host selection timeout","attr":{"replicaSet":"cfg","error":"FailedToSatisfyReadPreference: Could not find host matching read preference { mode: \"primary\" } for set cfg"}}

Below errors seen in Replica Set nodes in Cluster-2:

{"t":{"$date":"2025-03-25T09:58:26.350+00:00"},"s":"I",  "c":"ACCESS",   "id":5286307, "ctx":"conn59823","msg":"Failed to authenticate","attr":{"client":"127.0.0.6:57589","isSpeculative":false,"isClusterMember":false,"mechanism":"MONGODB-X509","user":"CN=cluster-1,O=PSMDB","db":"$external","error":"AuthenticationFailed: The provided certificate can only be used for cluster authentication, not client authentication. The current configuration does not allow x.509 cluster authentication, check the --clusterAuthMode flag","result":18,"metrics":{"conversation_duration":{"micros":81,"summary":{}}},"extraInfo":{}}}
{"t":{"$date":"2025-03-25T09:58:26.365+00:00"},"s":"I",  "c":"NETWORK",  "id":6723804, "ctx":"conn59826","msg":"Ingress TLS handshake complete","attr":{"durationMillis":378}}
{"t":{"$date":"2025-03-25T09:58:26.365+00:00"},"s":"W",  "c":"NETWORK",  "id":23236,   "ctx":"conn59826","msg":"Client connecting with server's own TLS certificate"}
{"t":{"$date":"2025-03-25T09:58:26.365+00:00"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn59826","msg":"client metadata","attr":{"remote":"127.0.0.6:35627","client":"conn59826","negotiatedCompressors":["snappy","zstd","zlib"],"doc":{"driver":{"name":"NetworkInterfaceTL-ReplicaSetMonitor-TaskExecutor","version":"7.0.15-9"},"os":{"type":"Linux","name":"Oracle Linux Server release 8.10","architecture":"x86_64","version":"Kernel 6.6.72+"}}}}
{"t":{"$date":"2025-03-25T09:58:26.464+00:00"},"s":"I",  "c":"ACCESS",   "id":5286307, "ctx":"conn59824","msg":"Failed to authenticate","attr":{"client":"127.0.0.6:33049","isSpeculative":false,"isClusterMember":false,"mechanism":"MONGODB-X509","user":"CN=cluster-1,O=PSMDB","db":"$external","error":"AuthenticationFailed: The provided certificate can only be used for cluster authentication, not client authentication. The current configuration does not allow x.509 cluster authentication, check the --clusterAuthMode flag","result":18,"metrics":{"conversation_duration":{"micros":77,"summary":{}}},"extraInfo":{}}}
{"t":{"$date":"2025-03-25T09:58:26.471+00:00"},"s":"I",  "c":"ACCESS",   "id":5286307, "ctx":"conn59825","msg":"Failed to authenticate","attr":{"client":"127.0.0.6:50501","isSpeculative":false,"isClusterMember":false,"mechanism":"MONGODB-X509","user":"CN=cluster-1,O=PSMDB","db":"$external","error":"AuthenticationFailed: The provided certificate can only be used for cluster authentication, not client authentication. The current configuration does not allow x.509 cluster authentication, check the --clusterAuthMode flag","result":18,"metrics":{"conversation_duration":{"micros":85,"summary":{}}},"extraInfo":{}}}

Hi @Slava_Sarzhan ,

Any pointers here?

Hi, I have reproduced the issue with the cert-manager. The problem is --enable-certificate-owner-ref=true option. Operator can’t use certificates with cert-manager owner-ref.

About your deployment. Are you sure that the certs were generated correctly? Please recheck doc: Splitting replica set across multiple data centers - Percona Operator for MongoDB

Thanks @Slava_Sarzhan for checking the cert-manager.
Is this Bug or need some documentation changes??

And regd deployment, certs were generated following the Percona documentation only.

After going through couple of MongoDB articles around this , I am suspecting issue is with respect to default authentication type used as KeyFile and MongoDB is expecting it to be X509 type authentication here for internal communication.

I could not find much documentation around creating & using X509 type auth in Percona documentation. I may be missing something here.

Below is the snippet for reference which was used to generate the certificates.

CLUSTER_NAME=cluster-1
CLUSTER_NAME_2=cluster-2
$ cat <<EOF | cfssl gencert -ca=ca.pem  -ca-key=ca-key.pem -config=./ca-config.json - | cfssljson -bare server
  {
    "hosts": [
      "localhost",
      "${CLUSTER_NAME}-rs0",
      "${CLUSTER_NAME}-rs0.${NAMESPACE}",
      "${CLUSTER_NAME}-rs0.${NAMESPACE}.svc.cluster.local",
      "*.${CLUSTER_NAME}-rs0",
      "*.${CLUSTER_NAME}-rs0.${NAMESPACE}",
      "*.${CLUSTER_NAME}-rs0.${NAMESPACE}.svc.cluster.local",
      "${CLUSTER_NAME}-rs0-0",
      "${CLUSTER_NAME}-rs0-0.${NAMESPACE}",
      "${CLUSTER_NAME}-rs0-0.${NAMESPACE}.svc.cluster.local",
      "*.${CLUSTER_NAME}-rs0-0",
      "*.${CLUSTER_NAME}-rs0-0.${NAMESPACE}",
      "*.${CLUSTER_NAME}-rs0-0.${NAMESPACE}.svc.cluster.local",
      "${CLUSTER_NAME}-rs0-1",
      "${CLUSTER_NAME}-rs0-1.${NAMESPACE}",
      "${CLUSTER_NAME}-rs0-1.${NAMESPACE}.svc.cluster.local",
      "*.${CLUSTER_NAME}-rs0-1",
      "*.${CLUSTER_NAME}-rs0-1.${NAMESPACE}",
      "*.${CLUSTER_NAME}-rs0-1.${NAMESPACE}.svc.cluster.local",
      "${CLUSTER_NAME}-rs0-2",
      "${CLUSTER_NAME}-rs0-2.${NAMESPACE}",
      "${CLUSTER_NAME}-rs0-2.${NAMESPACE}.svc.cluster.local",
      "*.${CLUSTER_NAME}-rs0-2",
      "*.${CLUSTER_NAME}-rs0-2.${NAMESPACE}",
      "*.${CLUSTER_NAME}-rs0-2.${NAMESPACE}.svc.cluster.local",
      "${CLUSTER_NAME}-mongos",
      "${CLUSTER_NAME}-mongos.${NAMESPACE}",
      "${CLUSTER_NAME}-mongos.${NAMESPACE}.svc.cluster.local",
      "*.${CLUSTER_NAME}-mongos",
      "*.${CLUSTER_NAME}-mongos.${NAMESPACE}",
      "*.${CLUSTER_NAME}-mongos.${NAMESPACE}.svc.cluster.local",
      "${CLUSTER_NAME}-mongos-*",
      "${CLUSTER_NAME}-mongos-*.${NAMESPACE}",
      "${CLUSTER_NAME}-mongos-*.${NAMESPACE}.svc.cluster.local",
      "*.${CLUSTER_NAME}-mongos-*",
      "*.${CLUSTER_NAME}-mongos-*.${NAMESPACE}",
      "*.${CLUSTER_NAME}-mongos-*.${NAMESPACE}.svc.cluster.local",
      "${CLUSTER_NAME}-cfg",
      "${CLUSTER_NAME}-cfg.${NAMESPACE}",
      "${CLUSTER_NAME}-cfg.${NAMESPACE}.svc.cluster.local",
      "*.${CLUSTER_NAME}-cfg",
      "*.${CLUSTER_NAME}-cfg.${NAMESPACE}",
      "*.${CLUSTER_NAME}-cfg.${NAMESPACE}.svc.cluster.local",
      "${CLUSTER_NAME}-cfg-0",
      "${CLUSTER_NAME}-cfg-0.${NAMESPACE}",
      "${CLUSTER_NAME}-cfg-0.${NAMESPACE}.svc.cluster.local",
      "*.${CLUSTER_NAME}-cfg-0",
      "*.${CLUSTER_NAME}-cfg-0.${NAMESPACE}",
      "*.${CLUSTER_NAME}-cfg-0.${NAMESPACE}.svc.cluster.local",
      "${CLUSTER_NAME}-cfg-1",
      "${CLUSTER_NAME}-cfg-1.${NAMESPACE}",
      "${CLUSTER_NAME}-cfg-1.${NAMESPACE}.svc.cluster.local",
      "*.${CLUSTER_NAME}-cfg-1",
      "*.${CLUSTER_NAME}-cfg-1.${NAMESPACE}",
      "*.${CLUSTER_NAME}-cfg-1.${NAMESPACE}.svc.cluster.local",
      "${CLUSTER_NAME}-cfg-2",
      "${CLUSTER_NAME}-cfg-2.${NAMESPACE}",
      "${CLUSTER_NAME}-cfg-2.${NAMESPACE}.svc.cluster.local",
      "*.${CLUSTER_NAME}-cfg-2",
      "*.${CLUSTER_NAME}-cfg-2.${NAMESPACE}",
      "*.${CLUSTER_NAME}-cfg-2.${NAMESPACE}.svc.cluster.local",
      "${CLUSTER_NAME_2}-rs0",
      "${CLUSTER_NAME_2}-rs0.${NAMESPACE_2}",
      "${CLUSTER_NAME_2}-rs0.${NAMESPACE_2}.svc.cluster.local",
      "*.${CLUSTER_NAME_2}-rs0",
      "*.${CLUSTER_NAME_2}-rs0.${NAMESPACE_2}",
      "*.${CLUSTER_NAME_2}-rs0.${NAMESPACE_2}.svc.cluster.local",
      "${CLUSTER_NAME_2}-rs0-0",
      "${CLUSTER_NAME_2}-rs0-0.${NAMESPACE_2}",
      "${CLUSTER_NAME_2}-rs0-0.${NAMESPACE_2}.svc.cluster.local",
      "*.${CLUSTER_NAME_2}-rs0-0",
      "*.${CLUSTER_NAME_2}-rs0-0.${NAMESPACE_2}",
      "*.${CLUSTER_NAME_2}-rs0-0.${NAMESPACE_2}.svc.cluster.local",
      "${CLUSTER_NAME_2}-rs0-1",
      "${CLUSTER_NAME_2}-rs0-1.${NAMESPACE_2}",
      "${CLUSTER_NAME_2}-rs0-1.${NAMESPACE_2}.svc.cluster.local",
      "*.${CLUSTER_NAME_2}-rs0-1",
      "*.${CLUSTER_NAME_2}-rs0-1.${NAMESPACE_2}",
      "*.${CLUSTER_NAME_2}-rs0-1.${NAMESPACE_2}.svc.cluster.local",
      "${CLUSTER_NAME_2}-rs0-2",
      "${CLUSTER_NAME_2}-rs0-2.${NAMESPACE_2}",
      "${CLUSTER_NAME_2}-rs0-2.${NAMESPACE_2}.svc.cluster.local",
      "*.${CLUSTER_NAME_2}-rs0-2",
      "*.${CLUSTER_NAME_2}-rs0-2.${NAMESPACE_2}",
      "*.${CLUSTER_NAME_2}-rs0-2.${NAMESPACE_2}.svc.cluster.local",
      "${CLUSTER_NAME_2}-mongos",
      "${CLUSTER_NAME_2}-mongos.${NAMESPACE_2}",
      "${CLUSTER_NAME_2}-mongos.${NAMESPACE_2}.svc.cluster.local",
      "*.${CLUSTER_NAME_2}-mongos",
      "*.${CLUSTER_NAME_2}-mongos.${NAMESPACE_2}",
      "*.${CLUSTER_NAME_2}-mongos.${NAMESPACE_2}.svc.cluster.local",
      "${CLUSTER_NAME_2}-mongos-*",
      "${CLUSTER_NAME_2}-mongos-*.${NAMESPACE_2}",
      "${CLUSTER_NAME_2}-mongos-*.${NAMESPACE_2}.svc.cluster.local",
      "*.${CLUSTER_NAME_2}-mongos-*",
      "*.${CLUSTER_NAME_2}-mongos-*.${NAMESPACE_2}",
      "*.${CLUSTER_NAME_2}-mongos-*.${NAMESPACE_2}.svc.cluster.local",
      "${CLUSTER_NAME_2}-cfg",
      "${CLUSTER_NAME_2}-cfg.${NAMESPACE_2}",
      "${CLUSTER_NAME_2}-cfg.${NAMESPACE_2}.svc.cluster.local",
      "*.${CLUSTER_NAME_2}-cfg",
      "*.${CLUSTER_NAME_2}-cfg.${NAMESPACE_2}",
      "*.${CLUSTER_NAME_2}-cfg.${NAMESPACE_2}.svc.cluster.local",
      "${CLUSTER_NAME_2}-cfg-0",
      "${CLUSTER_NAME_2}-cfg-0.${NAMESPACE_2}",
      "${CLUSTER_NAME_2}-cfg-0.${NAMESPACE_2}.svc.cluster.local",
      "*.${CLUSTER_NAME_2}-cfg-0",
      "*.${CLUSTER_NAME_2}-cfg-0.${NAMESPACE_2}",
      "*.${CLUSTER_NAME_2}-cfg-0.${NAMESPACE_2}.svc.cluster.local",
      "${CLUSTER_NAME_2}-cfg-1",
      "${CLUSTER_NAME_2}-cfg-1.${NAMESPACE_2}",
      "${CLUSTER_NAME_2}-cfg-1.${NAMESPACE_2}.svc.cluster.local",
      "*.${CLUSTER_NAME_2}-cfg-1",
      "*.${CLUSTER_NAME_2}-cfg-1.${NAMESPACE_2}",
      "*.${CLUSTER_NAME_2}-cfg-1.${NAMESPACE_2}.svc.cluster.local",
      "${CLUSTER_NAME_2}-cfg-2",
      "${CLUSTER_NAME_2}-cfg-2.${NAMESPACE_2}",
      "${CLUSTER_NAME_2}-cfg-2.${NAMESPACE_2}.svc.cluster.local",
      "*.${CLUSTER_NAME_2}-cfg-2",
      "*.${CLUSTER_NAME_2}-cfg-2.${NAMESPACE_2}",
      "*.${CLUSTER_NAME_2}-cfg-2.${NAMESPACE_2}.svc.cluster.local"
    ],
    "names": [
      {
        "O": "PSMDB"
      }
    ],
    "CN": "${CLUSTER_NAME/-rs0}",
    "key": {
      "algo": "rsa",
      "size": 2048
    }
  }
EOF
$ cfssl bundle -ca-bundle=ca.pem -cert=server.pem | cfssljson -bare server

$ kubectl create secret generic cluster-1-ssl-internal --from-file=tls.crt=server.pem --from-file=tls.key=server-key.pem --from-file=ca.crt=ca.pem --type=kubernetes.io/tls

I have created task to fix it Jira

Thanks @Slava_Sarzhan .

Is there any workaround for this cert manager issue? We are actually planning to install MongoDB using Percona Operator in Multi Cluster environment.

We are using cert-manager in our deployment and cannot get rid of it.

@surajm for x509 auth, you can check this blog post: Authenticating Your Clients to MongoDB on Kubernetes Using x509 Certificates