Mongo doesn't work

MongoDB Percona Operator for MongoDB
1

Description:

Mongo is installed in the Kubernetes cluster through the percona operator. During the night, the sl certificates expired and the cert-manager issued new ones. But Mongo stopped working with the new certificates.

Steps to Reproduce:

[Step-by-step instructions on how to reproduce the issue, including any specific settings or configurations]

Version:

operator 1.5
mongo 6.0.9-7

Logs:

log in mongos:
Liveness probe errored: command “/opt/percona/mongodb-healthcheck k8s liveness --component mongos --ssl --sslInsecure --sslCAFile /etc/mongodb-ssl/ca.crt --sslPEMKeyFile /tmp/tls.pem --startupDelaySeconds 10” timed out

Expected Result:

[What the user expected to see or happen before the issue occurred]

Actual Result:

[What actually happened when the user encountered the issue]

Additional Information:

[Include any additional information that could be helpful to diagnose the issue, such as browser or device information]

2

when trying to connect bypassing Mongos from PyCharm

Timed out after 10000 ms while waiting to connect. Client view of cluster state is {type=UNKNOWN, servers=[{address=10.128.0.129:27017, type=REPLICA_SET_GHOST, roundTripTime=189.4 ms, state=CONNECTED}, {address=10.128.0.169:27017, type=REPLICA_SET_GHOST, roundTripTime=183.0 ms, state=CONNECTED}, {address=10.128.0.168:27017, type=REPLICA_SET_GHOST, roundTripTime=189.4 ms, state=CONNECTED}, {address=10.128.0.141:27017, type=REPLICA_SET_GHOST, roundTripTime=194.1 ms, state=CONNECTED}].

3

ERROR in operator

2024-02-15T17:29:39.749Z	ERROR	Reconciler error	{"controller": "psmdb-controller", "object": {"name":"mongo-percona","namespace":"mongo-percona"}, "namespace": "mongo-percona", "name": "mongo-percona", "reconcileID": "0c1c6bb7-2ed6-4485-9beb-8a6276573fa7", "error": "reconcile StatefulSet for cfg: failed to run smartUpdate: failed to check active jobs: getting PBM object: create PBM connection to mongo-percona-rs0-0.mongo-percona-rs0.mongo-percona.svc.cluster.local:27017,mongo-percona-rs0-1.mongo-percona-rs0.mongo-percona.svc.cluster.local:27017,mongo-percona-rs0-2.mongo-percona-rs0.mongo-percona.svc.cluster.local:27017,mongo-percona-rs0-3.mongo-percona-rs0.mongo-percona.svc.cluster.local:27017,mongo-percona-rs0-arbiter-0.mongo-percona-rs0.mongo-percona.svc.cluster.local:27017: create mongo connection: mongo ping: server selection error: server selection timeout, current topology: { Type: Unknown, Servers: [{ Addr: mongo-percona-rs0-0.mongo-percona-rs0.mongo-percona.svc.cluster.local:27017, Type: RSGhost, Average RTT: 1386127 }, { Addr: mongo-percona-rs0-1.mongo-percona-rs0.mongo-percona.svc.cluster.local:27017, Type: RSGhost, Average RTT: 1472343 }, { Addr: mongo-percona-rs0-2.mongo-percona-rs0.mongo-percona.svc.cluster.local:27017, Type: RSGhost, Average RTT: 1215980 }, { Addr: mongo-percona-rs0-3.mongo-percona-rs0.mongo-percona.svc.cluster.local:27017, Type: RSGhost, Average RTT: 1147438 }, { Addr: mongo-percona-rs0-arbiter-0.mongo-percona-rs0.mongo-percona.svc.cluster.local:27017, Type: RSGhost, Average RTT: 1087948 }, ] }", "errorVerbose": "reconcile StatefulSet for cfg: failed to run smartUpdate: failed to check active jobs: getting PBM object: create PBM connection to mongo-percona-rs0-0.mongo-percona-rs0.mongo-percona.svc.cluster.local:27017,mongo-percona-rs0-1.mongo-percona-rs0.mongo-percona.svc.cluster.local:27017,mongo-percona-rs0-2.mongo-percona-rs0.mongo-percona.svc.cluster.local:27017,mongo-percona-rs0-3.mongo-percona-rs0.mongo-percona.svc.cluster.local:27017,mongo-percona-rs0-arbiter-0.mongo-percona-rs0.mongo-percona.svc.cluster.local:27017: create mongo connection: mongo ping: server selection error: server selection timeout, current topology: { Type: Unknown, Servers: [{ Addr: mongo-percona-rs0-0.mongo-percona-rs0.mongo-percona.svc.cluster.local:27017, Type: RSGhost, Average RTT: 1386127 }, { Addr: mongo-percona-rs0-1.mongo-percona-rs0.mongo-percona.svc.cluster.local:27017, Type: RSGhost, Average RTT: 1472343 }, { Addr: mongo-percona-rs0-2.mongo-percona-rs0.mongo-percona.svc.cluster.local:27017, Type: RSGhost, Average RTT: 1215980 }, { Addr: mongo-percona-rs0-3.mongo-percona-rs0.mongo-percona.svc.cluster.local:27017, Type: RSGhost, Average RTT: 1147438 }, { Addr: mongo-percona-rs0-arbiter-0.mongo-percona-rs0.mongo-percona.svc.cluster.local:27017, Type: RSGhost, Average RTT: 1087948 }, ] }\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:423\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.1/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.1/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.1/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.1/pkg/internal/controller/controller.go:227\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1598"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.1/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.1/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.1/pkg/internal/controller/controller.go:227