Hi,
I have cross site replication enabled using percona mongodb operator, mongo version 6.0.4-3
we are using oracle load balancer to expose the pods and we noticed a drop happens suddenly in the connection and when we check the logs we found the below:
{"t":{"$date":"2023-09-12T19:15:01.043+00:00"},"s":"I", "c":"CONNPOOL", "id":22572, "ctx":"ShardRegistry","msg":"Dropping all pooled connections","attr":{"hostAndPort":"10.11.10.18:27017","error":"ConnectionPoolExpired: Pool for 10.11.10.18:27017 has expired."}}
{"t":{"$date":"2023-09-12T19:19:31.387+00:00"},"s":"I", "c":"CONNPOOL", "id":22572, "ctx":"ShardRegistry","msg":"Dropping all pooled connections","attr":{"hostAndPort":"10.11.10.1:27017","error":"ConnectionPoolExpired: Pool for 10.11.10.1:27017 has expired."}}
please note the IPs mentioned in the logs is for the DR site.
here the graph from the LB:
I noticed the drop happened every 2 hours also number of connections varies from 3K to 5K
appreciate your usual support
can see this from pmm server
also please check the below output from mongostat the number of connection was 5816 and suddenly became 851
insert query update delete getmore command dirty used flushes vsize res qrw arw net_in net_out conn set repl time
*0 *0 *0 *0 0 462|0 0.0% 81.6% 0 13.5G 4.73G 0|0 2|0 101k 552k 5809 rs0 SLV Sep 12 13:04:36.247
*0 *0 *0 *0 0 566|0 0.0% 81.3% 0 13.5G 4.73G 0|0 2|0 137k 673k 5809 rs0 SLV Sep 12 13:04:37.241
*0 *0 *0 *0 0 1500|0 0.0% 80.7% 0 13.5G 4.73G 0|0 0|0 436k 35.3m 5812 rs0 SLV Sep 12 13:04:38.238
*0 *0 *0 *0 3 237|0 0.0% 81.3% 0 13.5G 4.73G 0|0 2|0 66.1k 440k 5812 rs0 SLV Sep 12 13:04:39.241
*0 3 *0 *0 0 624|0 0.0% 80.9% 0 13.4G 4.73G 0|0 2|0 146k 979k 5789 rs0 SLV Sep 12 13:04:40.245
*0 *0 *0 *0 0 504|0 0.0% 81.1% 0 13.5G 4.73G 0|0 2|0 111k 590k 5810 rs0 SLV Sep 12 13:04:41.244
*0 *0 *0 *0 0 246|0 0.0% 81.2% 0 13.5G 4.73G 0|0 2|0 57.0k 336k 5813 rs0 SLV Sep 12 13:04:42.244
*0 *0 *0 *0 0 233|0 0.0% 80.8% 0 13.5G 4.73G 0|0 2|0 61.1k 307k 5816 rs0 SLV Sep 12 13:04:43.241
*0 *0 *0 *0 3 287|0 0.0% 80.8% 0 13.5G 4.73G 0|0 2|0 303k 566k 5816 rs0 SLV Sep 12 13:04:44.241
*0 2 *0 *0 0 70|0 0.0% 80.5% 0 13.0G 4.77G 0|0 2|0 20.1k 733k 851 rs0 SLV Sep 12 13:04:45.444
here sudden drop after it reaches 8.94K connections, this very weird isn’t it?!
I tried to get the number of connections using
var idleConnections = db.serverStatus().connections.available;
print("Number of idle connections: " + idleConnections);
and I got Number of idle connections: 837677
can someone please tell why mongo opens all of this idle connection after we applied cross site replication ?
@Sergey_Pronin
