Percona MonogDB Cluster Fails when enabling TLS

MongoDB Percona Operator for MongoDB
1

Description:

Hello, I am trying to deploy A Percona MongoDB cluster with TLS enabled (Integrated with cert-manager) but it isn’t working, I am deploying both the operator and the replicaset using the helm charts, here are my values

pause: false
unmanaged: false

updateStrategy: RollingUpdate
upgradeOptions:
  apply: Disabled

image:
  tag: 6.0.15

tls:
  mode: requireTLS
  certValidityDuration: 876000h
  allowInvalidCertificates: false
  issuerConf:
    name: selfsigned-cluster-issuer
    kind: ClusterIssuer

secrets:
  key: pilgrimage-rs
  users: pilgrimage-rs
  sse: pilgrimage-rs

sharding:
  enabled: false

replsets:
  rs0:
    name: pilgrimage-rs
    size: 3
    affinity:
      antiAffinityTopologyKey: "kubernetes.io/hostname"
    tolerations:
      - effect: NoSchedule
        key: stateful
        operator: Exists
    # podSecurityContext: {}
    # containerSecurityContext: {}
    nodeSelector:
      node-role.kubernetes.io/storage: "true"

    livenessProbe:
      failureThreshold: 4
      initialDelaySeconds: 60
      periodSeconds: 30
      timeoutSeconds: 10
      startupDelaySeconds: 120
    readinessProbe:
      failureThreshold: 8
      initialDelaySeconds: 10
      periodSeconds: 3
      successThreshold: 1
      timeoutSeconds: 2
    podDisruptionBudget:
      maxUnavailable: 1
    resources:
      limits:
        memory: "0.5G"
      requests:
        cpu: "300m"
        memory: "0.5G"
    volumeSpec:
      persistentVolumeClaim:
        storageClassName: vsphere-csi-ext4
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 3Gi

backup:
  enabled: false
  image:
    repository: percona/percona-backup-mongodb
    tag: 2.4.1
  # podSecurityContext: {}
  # containerSecurityContext: {}
  resources:
    limits:
      memory: 100Mi
    requests:
      cpu: 300m
      memory: 100Mi
  storages:
    minio:
      type: s3
      s3:
        region: dit
        bucket: percona-mongodb-dev
        credentialsSecret: pilgrimage-rs
        uploadPartSize: 10485760
        maxUploadParts: 10000
        retryer:
          numMaxRetries: 3
          minRetryDelay: 10ms
          maxRetryDelay: 5m
        endpointUrl: https://s3.dev.krd
        prefix: "pilgrimage-dev-"
        insecureSkipTLSVerify: false
  pitr:
    enabled: true
    oplogOnly: false
    oplogSpanMin: 10
    compressionType: gzip
    compressionLevel: 6
  configuration:
    restoreOptions:
      batchSize: 500
      numInsertionWorkers: 10
      numDownloadWorkers: 4
      maxDownloadBufferMb: 0
      downloadChunkMb: 32
  tasks:
    - name: daily-minio
      enabled: true
      schedule: "0 0 * * *"
      keep: 3
      storageName: minio
      compressionType: gzip

And the operators values

replicaCount: 1

image:
  tag: 1.16.0

watchAllNamespaces: true

podSecurityContext:
  runAsNonRoot: true
  runAsUser: 2
  runAsGroup: 2
  fsGroup: 2
  fsGroupChangePolicy: "OnRootMismatch"

securityContext:
  allowPrivilegeEscalation: false
  capabilities:
    drop:
      - ALL
  seccompProfile:
    type: RuntimeDefault

env:
  resyncPeriod: 5s

resources:
  limits:
    memory: 300Mi
  requests:
    cpu: 100m
    memory: 300Mi

logStructured: true
logLevel: "INFO"

Logs:

{"t":{"$date":"2024-05-30T14:14:55.133+00:00"},"s":"E",  "c":"NETWORK",  "id":23256,   "ctx":"conn22","msg":"SSL peer certificate validation failed","attr":{"error":"SSL peer certificate validation failed: self signed certificate"}}
{"t":{"$date":"2024-05-30T14:14:55.133+00:00"},"s":"I",  "c":"NETWORK",  "id":22988,   "ctx":"conn22","msg":"Error receiving request from client. Ending connection from remote","attr":{"error":{"code":141,"codeName":"SSLHandshakeFailed","errmsg":"SSL peer certificate validation failed: self signed certificate"},"remote":"10.244.7.115:53746","connectionId":22}}
{"t":{"$date":"2024-05-30T14:14:55.631+00:00"},"s":"E",  "c":"NETWORK",  "id":23256,   "ctx":"conn23","msg":"SSL peer certificate validation failed","attr":{"error":"SSL peer certificate validation failed: self signed certificate"}}
{"t":{"$date":"2024-05-30T14:14:55.631+00:00"},"s":"I",  "c":"NETWORK",  "id":22988,   "ctx":"conn23","msg":"Error receiving request from client. Ending connection from remote","attr":{"error":{"code":141,"codeName":"SSLHandshakeFailed","errmsg":"SSL peer certificate validation failed: self signed certificate"},"remote":"10.244.7.115:53760","connectionId":23}}
{"t":{"$date":"2024-05-30T14:14:56.132+00:00"},"s":"E",  "c":"NETWORK",  "id":23256,   "ctx":"conn24","msg":"SSL peer certificate validation failed","attr":{"error":"SSL peer certificate validation failed: self signed certificate"}}
{"t":{"$date":"2024-05-30T14:14:56.132+00:00"},"s":"I",  "c":"NETWORK",  "id":22988,   "ctx":"conn24","msg":"Error receiving request from client. Ending connection from remote","attr":{"error":{"code":141,"codeName":"SSLHandshakeFailed","errmsg":"SSL peer certificate validation failed: self signed certificate"},"remote":"10.244.7.115:53762","connectionId":24}}
{"t":{"$date":"2024-05-30T14:14:56.632+00:00"},"s":"E",  "c":"NETWORK",  "id":23256,   "ctx":"conn26","msg":"SSL peer certificate validation failed","attr":{"error":"SSL peer certificate validation failed: self signed certificate"}}
{"t":{"$date":"2024-05-30T14:14:56.632+00:00"},"s":"I",  "c":"NETWORK",  "id":22988,   "ctx":"conn26","msg":"Error receiving request from client. Ending connection from remote","attr":{"error":{"code":141,"codeName":"SSLHandshakeFailed","errmsg":"SSL peer certificate validation failed: self signed certificate"},"remote":"10.244.7.115:53776","connectionId":26}}
{"t":{"$date":"2024-05-30T14:14:57.132+00:00"},"s":"E",  "c":"NETWORK",  "id":23256,   "ctx":"conn27","msg":"SSL peer certificate validation failed","attr":{"error":"SSL peer certificate validation failed: self signed certificate"}}
{"t":{"$date":"2024-05-30T14:14:57.132+00:00"},"s":"I",  "c":"NETWORK",  "id":22988,   "ctx":"conn27","msg":"Error receiving request from client. Ending connection from remote","attr":{"error":{"code":141,"codeName":"SSLHandshakeFailed","errmsg":"SSL peer certificate validation failed: self signed certificate"},"remote":"10.244.7.115:49868","connectionId":27}}
{"t":{"$date":"2024-05-30T14:14:57.632+00:00"},"s":"E",  "c":"NETWORK",  "id":23256,   "ctx":"conn28","msg":"SSL peer certificate validation failed","attr":{"error":"SSL peer certificate validation failed: self signed certificate"}}
{"t":{"$date":"2024-05-30T14:14:57.632+00:00"},"s":"I",  "c":"NETWORK",  "id":22988,   "ctx":"conn28","msg":"Error receiving request from client. Ending connection from remote","attr":{"error":{"code":141,"codeName":"SSLHandshakeFailed","errmsg":"SSL peer certificate validation failed: self signed certificate"},"remote":"10.244.7.115:49884","connectionId":28}}
2

Hi, are you generating your own certificates? if using the default self-signed certs you will need to enable tls.allowInvalidCertificates. Have a look at Transport encryption (TLS/SSL) - Percona Operator for MongoDB

3

Hello, I managed to solve the issue, I assumed that I needed to generate the tls/ssl certs myself, turns out you just need to install cert-manager and percona will take care of the generation of the tls certs itself

4

glad to hear you sorted it out