I have mongodb server running in kubernetes as replicaset.
Mongodb server and the operator were installed using Helm charts from Percona. (I’ll add custom values that I specified during the installation at the end of this post)
This mongodb replicaset cluster is exposed using AWS nlb.
Each replicaset is accessible from NLB DNS name, with ssl=false flag in mongosh connection string.
Also I have Cert-Manager installed in the cluster and mongodb operator created self-signed certificates.
But when I try to connect to each replicaset using TLS certs I get this error
“MongoServerSelectionError: self-signed certificate”
I do not want to create trusted certs manually and would like to use selfsigned certificates.
What can be done to achieve this ?
I’m trying to move from MongoDB Community Operator/DB to Percona. But I have this and one more issue, which is related to exposing replicaset outside the kubernetes cluster.
Should I Open another Topic for this issue or write about it in this one ?
Here are configurations that I’m using for helm installation:
percona-mongodb-server helm chart values
helm install percona-mongodb-server percona/psmdb-db -n mongodb-percona -f db-values.yaml --version 1.14.3
finalizers:
- delete-psmdb-pods-in-order
clusterServiceDNSMode: "External"
nameOverride: ""
fullnameOverride: "mongo-database"
crVersion: 1.14.0
pause: false
unmanaged: false
allowUnsafeConfigurations: false
multiCluster:
enabled: false
updateStrategy: SmartUpdate
upgradeOptions:
versionServiceEndpoint: https://check.percona.com
apply: disabled
schedule: "0 2 * * *"
setFCV: false
image:
repository: percona/percona-server-mongodb
tag: 6.0.4-3
imagePullPolicy: Always
tls:
certValidityDuration: 2160h
pmm:
enabled: false
image:
repository: percona/pmm-client
tag: 2.35.0
serverHost: monitoring-service
replsets:
- name: rs0
size: 3
affinity:
antiAffinityTopologyKey: "kubernetes.io/hostname"
podDisruptionBudget:
maxUnavailable: 1
expose:
enabled: true
exposeType: LoadBalancer
serviceAnnotations:
service.beta.kubernetes.io/aws-load-balancer-backend-protocol: http
service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: instance
service.beta.kubernetes.io/aws-load-balancer-scheme: internal
service.beta.kubernetes.io/aws-load-balancer-type: external
service.beta.kubernetes.io/aws-load-balancer-healthcheck-healthy-threshold: "3"
service.beta.kubernetes.io/aws-load-balancer-healthcheck-unhealthy-threshold: "3"
service.beta.kubernetes.io/aws-load-balancer-healthcheck-timeout: "10"
service.beta.kubernetes.io/aws-load-balancer-healthcheck-interval: "120"
service.beta.kubernetes.io/aws-load-balancer-target-group-attributes: preserve_client_ip.enabled=false
arbiter:
enabled: false
size: 1
affinity:
antiAffinityTopologyKey: "kubernetes.io/hostname"
resources:
limits:
# cpu: "300m"
memory: "0.5G"
requests:
cpu: "100m"
memory: "0.5G"
volumeSpec:
pvc:
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: 10Gi
sharding:
enabled: false
backup:
enabled: false
image:
repository: percona/percona-backup-mongodb
tag: 2.0.4
serviceAccountName: percona-server-mongodb-operator
storages:
s3-eu-central:
type: s3
s3:
bucket: percona-mongodb-backup
credentialsSecret: percona-mongo-backup-s3
region: eu-central-1
prefix: ""
uploadPartSize: 10485760
maxUploadParts: 10000
storageClass: STANDARD
insecureSkipTLSVerify: false
pitr:
enabled: false
tasks:
- name: daily-s3-eu-central
enabled: true
schedule: "0 0 * * *"
keep: 3
storageName: s3-eu-central
compressionType: gzip
for the operator, I just used default values, just changed the name :
helm install percona-mongodb-operator percona/psmdb-operator --set fullnameOverride=percona-mongodb-operator --version 1.14.2 --namespace mongodb-percona
I use mongosh client to connect to MongoDB:
mongosh \
--tls \
--tlsCAFile ca.crt \
--tlsCertificateKeyFile certificateKey.pem \
"mongodb://clusterAdmin:somerandomepassword@somerandome-nlb-name.elb.eu-central-1.amazonaws.com/admin?replicaSet=rs0"
even if I add “–tlsAllowInvalidCertificates” flag to mongosh, i get “MongoServerSelectionError: Server selection timed out after 30000 ms”
There are no error logs in operator, but in MongoDB server instances, I see this log being repeated ““c”:“NETWORK”, “id”:23235, “ctx”:“conn14196”,“msg”:“SSL peer certificate validation failed”,“attr”:{“reason”:“self signed certificate”}}”
Some more logs from Mongod
mongod {"t":{"$date":"2023-07-25T10:28:07.772+00:00"},"s":"I", "c":"NETWORK", "id":51800, "ctx":"conn14189","msg":"client metadata","attr":{"remote":"10.34.12.156:35210","client":"conn14189","doc":{"driver":{"name":"mongo-go-driver","version":"v1.11.2"},"os":{"type":"
linux","architecture":"amd64"},"platform":"go1.19.6"}}}
mongod {"t":{"$date":"2023-07-25T10:28:07.775+00:00"},"s":"W", "c":"NETWORK", "id":23235, "ctx":"conn14190","msg":"SSL peer certificate validation failed","attr":{"reason":"self signed certificate"}}
mongod {"t":{"$date":"2023-07-25T10:28:07.776+00:00"},"s":"I", "c":"NETWORK", "id":51800, "ctx":"conn14190","msg":"client metadata","attr":{"remote":"10.34.12.156:35216","client":"conn14190","doc":{"driver":{"name":"mongo-go-driver","version":"v1.11.2"},"os":{"type":"
linux","architecture":"amd64"},"platform":"go1.19.6"}}}
mongod {"t":{"$date":"2023-07-25T10:28:07.782+00:00"},"s":"I", "c":"-", "id":20883, "ctx":"conn14190","msg":"Interrupted operation as its client disconnected","attr":{"opId":130849}}
mongod {"t":{"$date":"2023-07-25T10:28:07.783+00:00"},"s":"I", "c":"NETWORK", "id":22944, "ctx":"conn14190","msg":"Connection ended","attr":{"remote":"10.34.12.156:35216","uuid":"b3fc1c4b-15ac-4984-8281-455850f41a33","connectionId":14190,"connectionCount":8}}
mongod {"t":{"$date":"2023-07-25T10:28:07.785+00:00"},"s":"I", "c":"NETWORK", "id":22944, "ctx":"conn14189","msg":"Connection ended","attr":{"remote":"10.34.12.156:35210","uuid":"2e63a2fb-cc98-4ed3-9279-25d6bbd61611","connectionId":14189,"connectionCount":7}}
mongod {"t":{"$date":"2023-07-25T10:28:07.811+00:00"},"s":"I", "c":"NETWORK", "id":22943, "ctx":"listener","msg":"Connection accepted","attr":{"remote":"10.34.18.7:59088","uuid":"9edd9ff2-4214-4c14-8b9e-bd696b72242e","connectionId":14191,"connectionCount":8}}
mongod {"t":{"$date":"2023-07-25T10:28:07.812+00:00"},"s":"I", "c":"NETWORK", "id":22943, "ctx":"listener","msg":"Connection accepted","attr":{"remote":"10.34.18.152:32319","uuid":"41641cab-2350-477e-be9e-325bc08e9967","connectionId":14192,"connectionCount":9}}
mongod {"t":{"$date":"2023-07-25T10:28:07.821+00:00"},"s":"W", "c":"NETWORK", "id":23235, "ctx":"conn14191","msg":"SSL peer certificate validation failed","attr":{"reason":"self signed certificate"}}
mongod {"t":{"$date":"2023-07-25T10:28:07.822+00:00"},"s":"I", "c":"NETWORK", "id":51800, "ctx":"conn14191","msg":"client metadata","attr":{"remote":"10.34.18.7:59088","client":"conn14191","doc":{"driver":{"name":"mongo-go-driver","version":"v1.11.2"},"os":{"type":"li
nux","architecture":"amd64"},"platform":"go1.19.6"}}}
mongod {"t":{"$date":"2023-07-25T10:28:07.825+00:00"},"s":"W", "c":"NETWORK", "id":23235, "ctx":"conn14192","msg":"SSL peer certificate validation failed","attr":{"reason":"self signed certificate"}}
mongod {"t":{"$date":"2023-07-25T10:28:07.832+00:00"},"s":"I", "c":"NETWORK", "id":51800, "ctx":"conn14192","msg":"client metadata","attr":{"remote":"10.34.18.152:32319","client":"conn14192","doc":{"driver":{"name":"mongo-go-driver","version":"v1.11.2"},"os":{"type":"
linux","architecture":"amd64"},"platform":"go1.19.6"}}}
mongod {"t":{"$date":"2023-07-25T10:28:07.874+00:00"},"s":"I", "c":"NETWORK", "id":22943, "ctx":"listener","msg":"Connection accepted","attr":{"remote":"10.34.12.156:35220","uuid":"ea6878f9-ed9e-491a-9bd9-68e4c7136bd3","connectionId":14193,"connectionCount":10}}
mongod {"t":{"$date":"2023-07-25T10:28:07.875+00:00"},"s":"I", "c":"NETWORK", "id":22943, "ctx":"listener","msg":"Connection accepted","attr":{"remote":"10.34.12.156:35236","uuid":"42032b15-ab79-4250-ab1c-c1cd58b914d9","connectionId":14194,"connectionCount":11}}
mongod {"t":{"$date":"2023-07-25T10:28:07.881+00:00"},"s":"W", "c":"NETWORK", "id":23235, "ctx":"conn14193","msg":"SSL peer certificate validation failed","attr":{"reason":"self signed certificate"}}
mongod {"t":{"$date":"2023-07-25T10:28:07.883+00:00"},"s":"I", "c":"NETWORK", "id":51800, "ctx":"conn14193","msg":"client metadata","attr":{"remote":"10.34.12.156:35220","client":"conn14193","doc":{"driver":{"name":"mongo-go-driver","version":"v1.11.2"},"os":{"type":"
linux","architecture":"amd64"},"platform":"go1.19.6"}}}
mongod {"t":{"$date":"2023-07-25T10:28:07.883+00:00"},"s":"W", "c":"NETWORK", "id":23235, "ctx":"conn14194","msg":"SSL peer certificate validation failed","attr":{"reason":"self signed certificate"}}
mongod {"t":{"$date":"2023-07-25T10:28:07.883+00:00"},"s":"I", "c":"NETWORK", "id":51800, "ctx":"conn14194","msg":"client metadata","attr":{"remote":"10.34.12.156:35236","client":"conn14194","doc":{"driver":{"name":"mongo-go-driver","version":"v1.11.2"},"os":{"type":"
linux","architecture":"amd64"},"platform":"go1.19.6"}}}
mongod {"t":{"$date":"2023-07-25T10:28:07.888+00:00"},"s":"I", "c":"-", "id":20883, "ctx":"conn14193","msg":"Interrupted operation as its client disconnected","attr":{"opId":130857}}
mongod {"t":{"$date":"2023-07-25T10:28:07.889+00:00"},"s":"I", "c":"NETWORK", "id":22944, "ctx":"conn14193","msg":"Connection ended","attr":{"remote":"10.34.12.156:35220","uuid":"ea6878f9-ed9e-491a-9bd9-68e4c7136bd3","connectionId":14193,"connectionCount":10}}
mongod {"t":{"$date":"2023-07-25T10:28:07.890+00:00"},"s":"I", "c":"NETWORK", "id":22944, "ctx":"conn14194","msg":"Connection ended","attr":{"remote":"10.34.12.156:35236","uuid":"42032b15-ab79-4250-ab1c-c1cd58b914d9","connectionId":14194,"connectionCount":9}}
mongod {"t":{"$date":"2023-07-25T10:28:07.911+00:00"},"s":"I", "c":"NETWORK", "id":22943, "ctx":"listener","msg":"Connection accepted","attr":{"remote":"10.34.18.114:11142","uuid":"5adbd7f0-2e6c-4883-b325-7f5792a528d6","connectionId":14195,"connectionCount":10}}
mongod {"t":{"$date":"2023-07-25T10:28:07.911+00:00"},"s":"I", "c":"NETWORK", "id":22943, "ctx":"listener","msg":"Connection accepted","attr":{"remote":"10.34.18.114:41802","uuid":"22fc8aa5-dc0e-4dbf-9358-474371ea8dff","connectionId":14196,"connectionCount":11}}
mongod {"t":{"$date":"2023-07-25T10:28:07.919+00:00"},"s":"W", "c":"NETWORK", "id":23235, "ctx":"conn14195","msg":"SSL peer certificate validation failed","attr":{"reason":"self signed certificate"}}
mongod {"t":{"$date":"2023-07-25T10:28:07.919+00:00"},"s":"I", "c":"NETWORK", "id":51800, "ctx":"conn14195","msg":"client metadata","attr":{"remote":"10.34.18.114:11142","client":"conn14195","doc":{"driver":{"name":"mongo-go-driver","version":"v1.11.2"},"os":{"type":"
linux","architecture":"amd64"},"platform":"go1.19.6"}}}
mongod {"t":{"$date":"2023-07-25T10:28:07.924+00:00"},"s":"W", "c":"NETWORK", "id":23235, "ctx":"conn14196","msg":"SSL peer certificate validation failed","attr":{"reason":"self signed certificate"}}
mongod {"t":{"$date":"2023-07-25T10:28:07.924+00:00"},"s":"I", "c":"NETWORK", "id":51800, "ctx":"conn14196","msg":"client metadata","attr":{"remote":"10.34.18.114:41802","client":"conn14196","doc":{"driver":{"name":"mongo-go-driver","version":"v1.11.2"},"os":{"type":"
linux","architecture":"amd64"},"platform":"go1.19.6"}}}
If I missed any critical info, please let me know and I’ll provide.