Member failed Kubernetes liveness check: get replsetGetStatus response: (Unauthorized) command replSetGetStatus requires authentication"
In a nutshell, cluster creates certs, but while liveness check appears to use them, it fails. Same result if cert_manager installed, either the version shown in Percona docs or current version.
- exec mongod --bind_ip_all --auth --dbpath=/data/db --port=27017 --replSet=rs0 --storageEngine=wiredTiger --relaxPermChecks --clusterAuthMode=x509 --enableEncryption --encryptionKeyFile=/etc/mongodb-encryption/encryption-key --wiredTigerIndexPrefixCompression=true --tlsMode preferTLS --tlsCertificateKeyFile /tmp/tls.pem --tlsAllowInvalidCertificates --tlsClusterFile /tmp/tls-internal.pem --tlsCAFile /etc/mongodb-ssl/ca.crt --tlsClusterCAFile /etc/mongodb-ssl-internal/ca.crt
{“t”:{“$date”:“2023-04-13T10:11:23.918+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:4915701, “ctx”:“-”,“msg”:“Initialized wire specification”,“attr”:{“spec”:{“incomingExternalClient”:{“minWireVersion”:0,“maxWireVersion”:17},“incomingInternalClient”:{“minWireVersion”:0,“maxWireVersion”:17},“outgoing”:{“minWireVersion”:6,“maxWireVersion”:17},“isInternalClient”:true}}}
{“t”:{“$date”:“2023-04-13T10:11:23.920+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:4913010, “ctx”:“-”,“msg”:“Certificate information”,“attr”:{“subject”:“O=PSMDB”,“issuer”:“O=Root CA”,“thumbprint”:“384C6E35CF35FDC7C55CFBCE501D4077615888C4”,“notValidBefore”:{“$date”:“2023-04-13T09:50:16.000Z”},“notValidAfter”:{“$date”:{“$numberLong”:“253402300799000”}},“keyFile”:“/tmp/tls.pem”,“type”:“Server”}}
{“t”:{“$date”:“2023-04-13T10:11:23.920+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:4913011, “ctx”:“-”,“msg”:“Certificate information”,“attr”:{“subject”:“O=PSMDB”,“issuer”:“O=Root CA”,“thumbprint”:“7B057B1008A9A6CE4B4D6D042A6A8A3503EC7A03”,“notValidBefore”:{“$date”:“2023-04-13T09:50:17.000Z”},“notValidAfter”:{“$date”:{“$numberLong”:“253402300799000”}},“keyFile”:“/tmp/tls-internal.pem”,“type”:“Cluster”}}
{“t”:{“$date”:“2023-04-13T10:11:23.920+00:00”},“s”:“I”, “c”:“CONTROL”, “id”:23285, “ctx”:“-”,“msg”:“Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols ‘none’”}
{“t”:{“$date”:“2023-04-13T10:19:38.251+00:00”},“s”:“I”, “c”:“CONTROL”, “id”:23285, “ctx”:“main”,“msg”:“Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols ‘none’”}
{“t”:{“$date”:“2023-04-13T10:19:38.253+00:00”},“s”:“I”, “c”:“CONTROL”, “id”:23403, “ctx”:“initandlisten”,“msg”:“Build Info”,“attr”:{“buildInfo”:{“version”:“6.0.4-3”,“gitVersion”:“6c7d07d27d493392d5e4933b1173960fe97a5381”,“openSSLVersion”:“OpenSSL 1.1.1k FIPS 25 Mar 2021”,“modules”:,“allocator”:“tcmalloc”,“environment”:{“distarch”:“x86_64”,“target_arch”:“x86_64”}}}}
{“t”:{“$date”:“2023-04-13T10:19:38.254+00:00”},“s”:“I”, “c”:“CONTROL”, “id”:21951, “ctx”:“initandlisten”,“msg”:“Options set by command line”,“attr”:{“options”:{“net”:{“bindIp”:“*”,“port”:27017,“tls”:{“CAFile”:“/etc/mongodb-ssl/ca.crt”,“allowInvalidCertificates”:true,“certificateKeyFile”:“/tmp/tls.pem”,“clusterCAFile”:“/etc/mongodb-ssl-internal/ca.crt”,“clusterFile”:“/tmp/tls-internal.pem”,“mode”:“preferTLS”}},“replication”:{“replSet”:“rs0”},“security”:{“authorization”:“enabled”,“clusterAuthMode”:“x509”,“enableEncryption”:true,“encryptionKeyFile”:“/etc/mongodb-encryption/encryption-key”,“relaxPermChecks”:true},“storage”:{“dbPath”:“/data/db”,“engine”:“wiredTiger”,“wiredTiger”:{“indexConfig”:{“prefixCompression”:true}}}}}}
{“t”:{“$date”:“2023-04-13T10:19:38.957+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:23016, “ctx”:“listener”,“msg”:“Waiting for connections”,“attr”:{“port”:27017,“ssl”:“on”}}
{“t”:{“$date”:“2023-04-13T10:20:07.265+00:00”},“s”:“W”, “c”:“NETWORK”, “id”:23235, “ctx”:“conn14”,“msg”:“SSL peer certificate validation failed”,“attr”:{“reason”:“certificate signature failure”}}
{“t”:{“$date”:“2023-04-13T10:20:07.294+00:00”},“s”:“W”, “c”:“NETWORK”, “id”:23235, “ctx”:“conn16”,“msg”:“SSL peer certificate validation failed”,“attr”:{“reason”:“certificate signature failure”}}
Liveness: exec [/opt/percona/mongodb-healthcheck k8s liveness --ssl --sslInsecure --sslCAFile /etc/mongodb-ssl/ca.crt --sslPEMKeyFile /tmp/tls.pem --startupDelaySeconds 7200] delay=60s timeout=10s period=30s #success=1 #failure=4
Readiness: tcp-socket :27017 delay=10s timeout=2s period=3s #success=1 #failure=8
Warning Unhealthy 101s (x19 over 14m) kubelet (combined from similar events): Liveness probe failed: {“level”:“info”,“msg”:“Running Kubernetes liveness check for mongod”,“time”:“2023-04-13T10:10:52Z”}
{“level”:“error”,“msg”:“Member failed Kubernetes liveness check: get replsetGetStatus response: (Unauthorized) command replSetGetStatus requires authentication”,“time”:“2023-04-13T10:10:52Z”}
Log,pod and cluster creation info below.
Any suggestions as to issue with the certifcate please. I think I’ve seen something about changing the file location from /tmp, but not sure why this would help
also see this form of unathorised error
Warning Unhealthy 25m kubelet Liveness probe failed: {“level”:“info”,“msg”:“Running Kubernetes liveness check for mongod”,“time”:“2023-04-13T10:21:30Z”}
{“level”:“error”,“msg”:“Member failed Kubernetes liveness check: get oplog.rs info: (Unauthorized) not authorized on local to execute command { collStats: "oplog.rs", scale: 1073741824, lsid: { id: UUID("eee0da00-2514-445c-9509-16440a8ef39f") }, $clusterTime: { clusterTime: Timestamp(1681381289, 1), signature: { hash: BinData(0, 62317B8DF70AF7BA8AFC806AD835DB071A387B6D), keyId: 7221476003388850183 } }, $db: "local", $readPreference: { mode: "primaryPreferred" } }”,“time”:“2023-04-13T10:21:30Z”}
Finally, if i connect to the DB when its initially created,
root@kube-1:~# kubectl run -i --rm --tty percona-client --image=percona/percona-server-mongodb:5.0 --restart=Never – mongo “mongodb+srv://${ADMIN_USER}:${ADMIN_PASSWORD}@mongodb-clu1-psmdb-db-rs0.mongodb.svc.cluster.local/admin?replicaSet=rs0&ssl=false”
If you don’t see a command prompt, try pressing enter.
rs0:PRIMARY> rs.status()
{
“ok” : 0,
“errmsg” : “not authorized on admin to execute command { replSetGetStatus: 1.0, lsid: { id: UUID("0751e5db-977c-4765-95f4-420f9e51e242") }, $clusterTime: { clusterTime: Timestamp(1681383768, 1), signature: { hash: BinData(0, 05DD5220FD7EC1609700D9A9454750F857FA7E1C), keyId: 7221469835815813126 } }, $db: "admin" }”,
“code” : 13,
“codeName” : “Unauthorized”,
“$clusterTime” : {
“clusterTime” : Timestamp(1681383788, 1),
“signature” : {
“hash” : BinData(0,“Zp/MOY9jPpPAJ7oMqp0Ix/JYMBY=”),
“keyId” : NumberLong(“7221469835815813126”)
}
},
“operationTime” : Timestamp(1681383788, 1)
}
Thanks,
Mike