PMM prevents MongoDB from accepting connections randomly

Percona Monitoring and Management (PMM)
,
1

Sharing PMM agent errors reported.

● pmm-agent.service - pmm-agent
     Loaded: loaded (/usr/lib/systemd/system/pmm-agent.service; enabled; preset: disabled)
     Active: active (running) since Sun 2025-09-28 11:03:31 +04; 1 week 5 days ago
   Main PID: 462608 (pmm-agent)
      Tasks: 65 (limit: 100388)
     Memory: 221.4M
        CPU: 21h 15min 8.901s
     CGroup: /system.slice/pmm-agent.service
             ├─462608 /usr/sbin/pmm-agent --config-file=/usr/local/percona/pmm2/config/pmm-agent.yaml
             ├─462659 /usr/local/percona/pmm2/exporters/mongodb_exporter --collector.collstats-limit=200 --collector.diagnosticdata --collector.replicasetstatus --compatible-mode --discovering-mode --log.level=warn --mongodb.global-conn-pool --web.listen-address=:42000 --web.config=/usr/local/percona/pmm2/tmp/mongodb_exporter/agent_id/8ca0bcee-e266-466c-8e92-bf1c63a95012/webConfigPlaceholder
             ├─462673 /usr/local/percona/pmm2/exporters/node_exporter --collector.bonding --collector.buddyinfo --collector.cpu --collector.diskstats --collector.entropy --collector.filefd --collector.filesystem --collector.hwmon --collector.loadavg --collector.meminfo --collector.meminfo_numa --collector.netdev --collector.netstat "--collector.netstat.fields=^(.*_(InErrors|InErrs|InCsumErrors)|Tcp_(ActiveOpens|PassiveOpens|RetransSegs|CurrEstab|AttemptFails|OutSegs|InSegs|EstabResets|OutRsts|OutSegs)|Tcp_Rto(Algorithm|Min|Max)|Udp_(RcvbufErrors|SndbufErrors)|Udp(6?|Lite6?)_(InDatagrams|OutDatagrams|RcvbufErrors|SndbufErrors|NoPorts)|Icmp6?_(OutEchoReps|OutEchos|InEchos|InEchoReps|InAddrMaskReps|InAddrMasks|OutAddrMaskReps|OutAddrMasks|InTimestampReps|InTimestamps|OutTimestampReps|OutTimestamps|OutErrors|InDestUnreachs|OutDestUnreachs|InTimeExcds|InRedirects|OutRedirects|InMsgs|OutMsgs)|IcmpMsg_(InType3|OutTy>
             └─462683 /usr/local/percona/pmm2/exporters/vmagent -envflag.enable=true -envflag.prefix=VMAGENT_ -httpListenAddr=127.0.0.1:42002 -loggerLevel=INFO -promscrape.config=/usr/local/percona/pmm2/tmp/vm_agent/agent_id/d6a2ccda-25cd-416e-896f-ad1a7c17927d/vmagentscrapecfg -promscrape.maxScrapeSize=64MiB -remoteWrite.maxDiskUsagePerURL=1073741824 -remoteWrite.tlsInsecureSkipVerify=true -remoteWrite.tmpDataPath=/usr/local/percona/pmm2/tmp/vmagent-temp-dir -remoteWrite.url=https://172.150.0.56:443/victoriametrics/api/v1/write

Oct 09 20:02:30 xxxxxxxxxxxxxxxxxxxxxxxxxx-prod01.com pmm-agent[462608]: time="2025-10-09T20:02:30.426+04:00" level=error msg="couldn't create system.profile iterator, reason: server selection error: server selection timeout, current topology: { Type: Single, Servers: [{ Addr: 10.81.67.197:22000, Type: Unknown, Last error: connection(10.81.67.197:22000[-1615]) incomplete read of message header: read tcp 10.81.67.197:56518->10.81.67.197:22000: i/o timeout }, ] }" agentID=/agent_id/a954b85b-0045-440f-be70-e01272028a8b component=agent-builtin db=onlinefee type=qan_mongodb_profiler_agent
Oct 09 20:02:30 xxxxxxxxxxxxxxxxxxxxxxxxxx-prod01.com pmm-agent[462608]: time="2025-10-09T20:02:30.426+04:00" level=error msg="couldn't create system.profile iterator, reason: server selection error: server selection timeout, current topology: { Type: Single, Servers: [{ Addr: 10.81.67.197:22000, Type: Unknown, Last error: connection(10.81.67.197:22000[-1615]) incomplete read of message header: read tcp 10.81.67.197:56518->10.81.67.197:22000: i/o timeout }, ] }" agentID=/agent_id/a954b85b-0045-440f-be70-e01272028a8b component=agent-builtin db=db1 type=qan_mongodb_profiler_agent
Oct 09 20:02:30 xxxxxxxxxxxxxxxxxxxxxxxxxx-prod01.com pmm-agent[462608]: time="2025-10-09T20:02:30.465+04:00" level=error msg="couldn't create system.profile iterator, reason: server selection error: server selection timeout, current topology: { Type: Single, Servers: [{ Addr: 10.81.67.197:22000, Type: Unknown, Last error: connection(10.81.67.197:22000[-1615]) incomplete read of message header: read tcp 10.81.67.197:56518->10.81.67.197:22000: i/o timeout }, ] }" agentID=/agent_id/a954b85b-0045-440f-be70-e01272028a8b component=agent-builtin db=ALIZZDCMS_PRODUCT_CONFIG type=qan_mongodb_profiler_agent
Oct 09 20:02:30 xxxxxxxxxxxxxxxxxxxxxxxxxx-prod01.com pmm-agent[462608]: time="2025-10-09T20:02:30.465+04:00" level=error msg="couldn't create system.profile iterator, reason: server selection error: server selection timeout, current topology: { Type: Single, Servers: [{ Addr: 10.81.67.197:22000, Type: Unknown, Last error: connection(10.81.67.197:22000[-1615]) incomplete read of message header: read tcp 10.81.67.197:56518->10.81.67.197:22000: i/o timeout }, ] }" agentID=/agent_id/a954b85b-0045-440f-be70-e01272028a8b component=agent-builtin db=nioswitch_vtex type=qan_mongodb_profiler_agent
Oct 09 20:02:30 xxxxxxxxxxxxxxxxxxxxxxxxxx-prod01.com pmm-agent[462608]: time="2025-10-09T20:02:30.465+04:00" level=error msg="couldn't create system.profile iterator, reason: server selection error: server selection timeout, current topology: { Type: Single, Servers: [{ Addr: 10.81.67.197:22000, Type: Unknown, Last error: connection(10.81.67.197:22000[-1615]) incomplete read of message header: read tcp 10.81.67.197:56518->10.81.67.197:22000: i/o timeout }, ] }" agentID=/agent_id/a954b85b-0045-440f-be70-e01272028a8b component=agent-builtin db=debit-config-service type=qan_mongodb_profiler_agent
Oct 09 20:02:30 xxxxxxxxxxxxxxxxxxxxxxxxxx-prod01.com pmm-agent[462608]: time="2025-10-09T20:02:30.841+04:00" level=warning msg="couldn't retrieve data from cursor connection pool for 10.81.67.197:22000 was cleared because another operation failed with: connection(10.81.67.197:22000[-1615]) incomplete read of message header: read tcp 10.81.67.197:56518->10.81.67.197:22000: i/o timeout" agentID=/agent_id/a954b85b-0045-440f-be70-e01272028a8b component=agent-builtin db=itsp type=qan_mongodb_profiler_agent
Oct 09 20:02:30 xxxxxxxxxxxxxxxxxxxxxxxxxx-prod01.com pmm-agent[462608]: time="2025-10-09T20:02:30.841+04:00" level=warning msg="couldn't retrieve data from cursor connection pool for 10.81.67.197:22000 was cleared because another operation failed with: connection(10.81.67.197:22000[-1615]) incomplete read of message header: read tcp 10.81.67.197:56518->10.81.67.197:22000: i/o timeout" agentID=/agent_id/a954b85b-0045-440f-be70-e01272028a8b component=agent-builtin db=atm type=qan_mongodb_profiler_agent
Oct 09 20:02:30 xxxxxxxxxxxxxxxxxxxxxxxxxx-prod01.com pmm-agent[462608]: time="2025-10-09T20:02:30.841+04:00" level=warning msg="couldn't retrieve data from cursor connection pool for 10.81.67.197:22000 was cleared because another operation failed with: connection(10.81.67.197:22000[-1615]) incomplete read of message header: read tcp 10.81.67.197:56518->10.81.67.197:22000: i/o timeout" agentID=/agent_id/a954b85b-0045-440f-be70-e01272028a8b component=agent-builtin db=connect_mongodb_session_vapt_webapp type=qan_mongodb_profiler_agent
Oct 09 20:02:30 xxxxxxxxxxxxxxxxxxxxxxxxxx-prod01.com pmm-agent[462608]: time="2025-10-09T20:02:30.841+04:00" level=warning msg="couldn't retrieve data from cursor connection pool for 10.81.67.197:22000 was cleared because another operation failed with: connection(10.81.67.197:22000[-1615]) incomplete read of message header: read tcp 10.81.67.197:56518->10.81.67.197:22000: i/o timeout" agentID=/agent_id/a954b85b-0045-440f-be70-e01272028a8b component=agent-builtin db=dumy type=qan_mongodb_profiler_agent
Oct 09 20:02:30 xxxxxxxxxxxxxxxxxxxxxxxxxx-prod01.com pmm-agent[462608]: time="2025-10-09T20:02:30.842+04:00" level=warning msg="couldn't retrieve data from cursor connection pool for 10.81.67.197:22000 was cleared because another operation failed with: connection(10.81.67.197:22000[-1615]) incomplete read of message header: read tcp 10.81.67.197:56518->10.81.67.197:22000: i/o timeout" agentID=/agent_id/a954b85b-0045-440f-be70-e01272028a8b component=agent-builtin db=atmtxncbs type=qan_mongodb_profiler_agent
2

Hi,
if you have more then 100 DBs we recommend to use mongolog query source introduced in PMM 3.2.0 as query source for QAN.

Can be done with pmm-admin add mongodb –query-source=mongolog

1 Like
3

Thanks, will try that!

4

@nurlan

Will PMM 3 client not work with PMM 2 server?

Checking local pmm-agent status...
pmm-agent is running.
Registering pmm-agent on PMM Server...
Failed to register pmm-agent on PMM Server: Not Found.
5

Yes, it won’t work. There are a breaking changes.

6

Understood. Will upgrade both. Thanks again