We are facing a strange issue where we are not able to see one of the mongodb cluster on PMM2 dashboard. It’s a three node replicaset and I can see that all services/agents/nodes are successfully added on PMM Inventory List.
To have this cluster on PMM MongoDB, is there any other configuration needed on PMM dashboard? Or could there be some issue with mongodb-exporter? We are able to see all the three nodes on “OS/ Nodes overview” graph but not in MongoDB.
Any assistance is really appreciated!
Hi, try the metrics explorer to see if there are any metrics with the name mongodb. If not you have a problem either with the PMM user privileges or you didn’t run the pmm-admin add mongodb command.
Hope that helps!
Hi Ivan, Thank you for a kind response. I do see mongodb metrics under explorer for other clusters. It’s just this one particular cluster that is having issues where mongodb metrics not visible on pmm.
I checked the curl command for mongodb_exporter agent-ID locally on primary node, and I do see it is able to fetch mongodb metrics. Somehow these metrics are not getting displayed on pmm
$ pmm-admin list
Service type Service name Address and port Service ID
MongoDB <service_name> 127.0.0.1:27017 /service_id/c1b7f891-fc78-4894-94eb-6d6ace7ecf61
Agent type Status Metrics Mode Agent ID Service ID Port
pmm_agent Connected /agent_id/15e3b0bf-29d9-4b8e-9dbe-b9fb85d4c985 0
node_exporter Running push /agent_id/1868f8ea-cb1a-4f83-8e8d-c35f4808ceca 11001
mongodb_exporter Running push /agent_id/bcf098b8-73c1-4837-928a-4dac964ff3e3 /service_id/c1b7f891-fc78-4894-94eb-6d6ace7ecf61 11003
mongodb_profiler_agent Running /agent_id/73d9db6b-0187-41bd-ae61-6f3e80e0914b /service_id/c1b7f891-fc78-4894-94eb-6d6ace7ecf61 0
vmagent Running push /agent_id/3374d223-9764-4882-9fcb-ed5b28aa208d
$ time curl http://127.0.0.1:11003/metrics -u "pmm:/agent_id/bcf098b8-73c1-4837-928a-4dac964ff3e3"
. . . .
< getting all mongodb metrics here >
. . . .
Hello, if the exporter is showing the metrics locally, that would point to some connectivity issue to PMM server. Can you remove the client and install again? also make sure you use the same client version as PMM server.
We tried reinstalling the client still no luck with mongodb metrics. All our clusters are running below pmm server client version and they are running fine without any issue. It’s just this one cluster that has issues. I might be wrong but since OS/node metrics is working it shouldn’t be connectivity issue. I believe there should be some issue with mongodb-exporter
PMM Server: 2.31.0
pmm-admin version: 2.32.0
pmm-agent version: 2.32.0
Checking pmm-agent.log we intermittently see below error messages. Not sure if this can cause pmm to stop sending metrics to server?
INFO[2022-12-13T16:14:56.743+00:00] 2022-12-13T16:14:56.742Z error VictoriaMetrics/lib/promscrape/scrapework.go:355 error when scraping "http://127.0.0.1:11003/metrics" from job "mongodb_exporter_agent_id_bcf098b8-73c1-4837-928a-4dac964ff3e3_hr" . . . . . . . error when scraping "http://127.0.0.1:11003/metrics" with timeout 4.5s: timeout agentID=/agent_id/3374d223-9764-4882-9fcb-ed5b28aa208d component=agent-process type=vm_agent
Since we have large number of collections. After disabling collstats --disable-collectors=collstats we are no longer facing timeout errors. Dashboard is loading data as expected.