Logs flooded with ssl messages

MongoDB Percona Operator for MongoDB
1

Watching the logs of a fresh mongodb cluster, I see this constantly repeated

{"t":{"$date":"2022-04-07T21:16:59.236+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"172.16.109.243:44868","connectionId":1169946,"connectionCount":66}}
{"t":{"$date":"2022-04-07T21:16:59.242+00:00"},"s":"W",  "c":"NETWORK",  "id":23235,   "ctx":"conn1169946","msg":"SSL peer certificate validation failed","attr":{"reason":"certificate signature failure"}}
{"t":{"$date":"2022-04-07T21:16:59.242+00:00"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn1169946","msg":"client metadata","attr":{"remote":"172.16.109.243:44868","client":"conn1169946","doc":{"driver":{"name":"mongo-go-driver","version":"v1.7.0"},"os":{"type":"linux","architecture":"amd64"},"platform":"go1.17.5"}}}
{"t":{"$date":"2022-04-07T21:16:59.300+00:00"},"s":"I",  "c":"-",        "id":20883,   "ctx":"conn1169946","msg":"Interrupted operation as its client disconnected","attr":{"opId":20369024}}
{"t":{"$date":"2022-04-07T21:16:59.300+00:00"},"s":"I",  "c":"NETWORK",  "id":22944,   "ctx":"conn1169946","msg":"Connection ended","attr":{"remote":"172.16.109.243:44868","connectionId":1169946,"connectionCount":64}}

{"t":{"$date":"2022-04-07T21:30:25.417+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"172.16.86.217:40024","connectionId":1171393,"connectionCount":40}}
{"t":{"$date":"2022-04-07T21:30:25.417+00:00"},"s":"I",  "c":"NETWORK",  "id":23838,   "ctx":"conn1171393","msg":"SSL mode is set to 'preferred' and connection to remote is not using SSL.","attr":{"connectionId":1171393,"remote":"172.16.86.217:40024"}}
{"t":{"$date":"2022-04-07T21:30:25.418+00:00"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn1171393","msg":"client metadata","attr":{"remote":"172.16.86.217:40024","client":"conn1171393","doc":{"driver":{"name":"mongo-go-driver","version":"v1.7.0"},"os":{"type":"linux","architecture":"amd64"},"platform":"go1.17.5"}}}
{"t":{"$date":"2022-04-07T21:30:25.426+00:00"},"s":"I",  "c":"ACCESS",   "id":20250,   "ctx":"conn1171393","msg":"Authentication succeeded","attr":{"mechanism":"SCRAM-SHA-256","speculative":true,"principalName":"clusterMonitor","authenticationDatabase":"admin","remote":"172.16.86.217:40024","extraInfo":{}}}
{"t":{"$date":"2022-04-07T21:30:25.431+00:00"},"s":"I",  "c":"NETWORK",  "id":22944,   "ctx":"conn1171393","msg":"Connection ended","attr":{"remote":"172.16.86.217:40024","connectionId":1171393,"connectionCount":37}}
2 Likes
2

@dmn these are not critical ones and do not affect the operations of the cluster.
Do you feel we should decrease the amount of this noise?

1 Like
3

Well my brain reads SSL something in the logs and my first instinct is something has gone wrong. If they are normal, I may just see about filtering them before shipping the logs, or adjust the default log verbosity settings

2 Likes
4

Could you provide me a link with how I can disable these notices in logs? I have tried to find it out but I failed.
I have already disabled information logs on mongodb-mongos
in console mongo

use admin
db.auth('clusterAdmin','password');
db.setLogLevel(1);

But when I do pause: true → pause: false I have got the same LogLevel=0

2 Likes
5

I have found how

1 Like
6

In the end, It is not possible to disable just network information messages although you can disable all messages meantime Error and Debug messages thus it is not production mode.

1 Like
7

Hey @EXA_NIX ,

unfortunately this is the limitation coming from mongodb itself.

You can set any option described here: https://www.mongodb.com/docs/manual/reference/configuration-options/#core-options

You can do smth like this though:

  - name: rs0
...
    configuration: |
      systemLog:
        quiet: true
        component:
          network:
            verbosity: 3

This will set all the logs to quite mode, but keep high verbosity for network component. Or you can set other components verbosity levels as well.
See here: https://www.mongodb.com/docs/manual/reference/configuration-options/#systemlog.component-options

1 Like
8

Hello @Sergey_Pronin & @Ege_Gunes (since you commented on that here)

I was wondering if you could explain exactly what connects to the cluster with unsecure options? Is it the pmm-agent & the backup-agent and is there a reason why they can’t use secure options?

Wouldn’t it also be less taxing on the server to keep those connections open without closing and reconnecting them all the time? My guess is that those constant reconnections and the logs they incur probably do not speed up the server! It’s probably minimal of course, but it does also create a lot of bloat which might silence actual important logs.

Thanks!

9

Reviving this thread since I still believe it would be important to resolve the flooding in order to have important logs surfaced easily, while not having to turn on quiet mode (even if that one actually was silencing those connections logs) and still having the ability to detect if a connection keeps being closed and reopened, which might indicate an issue.

Wouldn’t it be possible for the Percona component(s) connecting to the deployment to keep a long-lived connection, and avoid those floods? Or is there implications in terms of performance?

Thanks @Sergey_Pronin!

Related post