QAN not showing up when adding a mongo node

[sgales]

Hey All,

I’ve added a node per MongoDB - Percona Monitoring and Management and I get metrics, but I never get any Query analytics for mongo nodes. It works great for mysql / rds, but doesn’t even show up for mongo.

I try something like this:

pmm-admin --server-url=https://<user>:<pass>@localhost --server-insecure-tls add mongodb --username=<mongo_user> --password=<mongo_pass> --agent-password=<different pass from mongo_pass> --query-source=profiler --environment=betamongo --cluster=test-mongo --replication-set=mongo-test --metrics-mode=auto --service-name=mongo-node1 --host=<fqdn of mongo node> --port=21337 --tls-skip-verify

It adds the host and I see agents running and I get metrics, but I never get any query analytics. I can’t even add a mongodb filter.

I’ve seen a few errors in the log around:
Cannot get node type to check if this is a mongos: connection(<node>:21337[-11]) failed to write: context canceled

But I don’t see anything explicitly failing, and the agent seems to be running, I just don’t see any query analytics and I am at a bit of a loss at this point.

Could I be mistaken and username/pass should be the pmm_mongdb user? And then just leave out the agent_password altogether?

Running on this version:
ProjectName: pmm-admin
Version: 2.25.0
PMMVersion: 2.25.0
Timestamp: 2021-12-13 09:38:36 (UTC)
FullCommit: 4b81157ad02975c417daef2600cb0dcd3907ffa1

Thanks!

[Ivan_Groenewold]

Hi, you should not need to provide the agent password to add the MongoDB Service. Only provide the mongodb user credentials that you created for pmm to connect to your mongodb. Also please double check that you have the profiler enabled and that the profile collection is being populated. If that is ok, it is most likely a permission issue.

[sgales]

Ok, I recreated the agent with just the pmm creds to monitor the instance.

1. Updated the agent to use the correct username and password
2. Profiling is enabled, e.g. show profile in the mongo shell works and returns data
3. I’ve logged into the mongo instance as pmm user/pmm password and can run show profile

So I don’t think its a permissions issue.

I see some errors in pmm-agent.log

INFO[2022-02-08T20:38:36.454+00:00] time="2022-02-08T20:38:36Z" level=error msg="Cannot get node type to check if this is a mongos: connection(mongo-host:21337[-506]) failed to write: context canceled"  agentID=/agent_id/3f5579d2-eea9-4eeb-b2a7-d328cd2c0c86 component=agent-process type=mongodb_exporter
INFO[2022-02-08T20:38:41.452+00:00] time="2022-02-08T20:38:41Z" level=error msg="cannot get replSetGetStatus: connection() error occured during connection handshake: auth error: sasl conversation error: unable to authenticate using mechanism \"SCRAM-SHA-256\": connection(mongo-host:21337[-507]) incomplete read of message header: context canceled"  agentID=/agent_id/3f5579d2-eea9-4eeb-b2a7-d328cd2c0c86 component=agent-process type=mongodb_exporter
INFO[2022-02-08T20:38:41.452+00:00] time="2022-02-08T20:38:41Z" level=error msg="Cannot get node type to check if this is a mongos: connection(mongo-host:21337[-506]) failed to write: context canceled"  agentID=/agent_id/3f5579d2-eea9-4eeb-b2a7-d328cd2c0c86 component=agent-process type=mongodb_exporter
INFO[2022-02-08T20:39:01.452+00:00] time="2022-02-08T20:39:01Z" level=error msg="cannot run getDiagnosticData: connection(mongo-host:21337[-509]) incomplete read of full message: context canceled"  agentID=/agent_id/3f5579d2-eea9-4eeb-b2a7-d328cd2c0c86 component=agent-process type=mongodb_exporter
INFO[2022-02-08T20:39:01.453+00:00] time="2022-02-08T20:39:01Z" level=error msg="Cannot get node type to check if this is a mongos: connection(mongo-host:21337[-506]) failed to write: context canceled"  agentID=/agent_id/3f5579d2-eea9-4eeb-b2a7-d328cd2c0c86 component=agent-process type=mongodb_exporter
INFO[2022-02-08T20:39:01.453+00:00] time="2022-02-08T20:39:01Z" level=error msg="cannot get replSetGetStatus: connection(mongo-host:21337[-506]) failed to write: context canceled"  agentID=/agent_id/3f5579d2-eea9-4eeb-b2a7-d328cd2c0c86 component=agent-process type=mongodb_exporter
INFO[2022-02-08T20:39:01.454+00:00] time="2022-02-08T20:39:01Z" level=error msg="cannot get replSetGetStatus: connection(mongo-host:21337[-506]) failed to write: context canceled"  agentID=/agent_id/3f5579d2-eea9-4eeb-b2a7-d328cd2c0c86 component=agent-process type=mongodb_exporter
INFO[2022-02-08T20:39:01.454+00:00] time="2022-02-08T20:39:01Z" level=error msg="cannot run getDiagnosticData: connection(mongo-host:21337[-508]) failed to write: context canceled"  agentID=/agent_id/3f5579d2-eea9-4eeb-b2a7-d328cd2c0c86 component=agent-process type=mongodb_exporter
INFO[2022-02-08T20:39:01.454+00:00] time="2022-02-08T20:39:01Z" level=error msg="error while checking mongodb connection: connection(mongo-host:21337[-506]) failed to write: context canceled. mongo_up is set to 0"  agentID=/agent_id/3f5579d2-eea9-4eeb-b2a7-d328cd2c0c86 component=agent-process type=mongodb_exporter
INFO[2022-02-08T20:39:01.454+00:00] time="2022-02-08T20:39:01Z" level=error msg="Cannot get node type to check if this is a mongos: connection(mongo-host:21337[-508]) failed to write: context canceled"  agentID=/agent_id/3f5579d2-eea9-4eeb-b2a7-d328cd2c0c86 component=agent-process type=mongodb_exporter
INFO[2022-02-08T21:36:31.453+00:00] time="2022-02-08T21:36:31Z" level=error msg="Cannot get node type to check if this is a mongos: connection(mongo-host:21337[-510]) failed to write: context canceled"  agentID=/agent_id/3f5579d2-eea9-4eeb-b2a7-d328cd2c0c86 component=agent-process type=mongodb_exporter
INFO[2022-02-08T22:30:57.508+00:00] time="2022-02-08T22:30:57Z" level=error msg="Cannot get node type to check if this is a mongos: connection(mongo-host:21337[-4]) failed to write: context canceled"  agentID=/agent_id/a163ba2b-a96f-470c-915d-4aa1f84966c6 component=agent-process type=mongodb_exporter

[pmm-host][me ~]$ nc -zv mongo-host 21337
Ncat: Version 7.50 ( https://nmap.org/ncat )
Ncat: Connected to <ip_address>:21337.
Ncat: 0 bytes sent, 0 bytes received in 0.07 seconds.

Is there a way to inspect if the QAN is even running?

[Ivan_Groenewold]

Hi, you can check if QAN is up by running pmm-admin status. mongodb_profiler_agent should be in running state.
From the output above it seems the agent is having issues connecting to mongodb with the provided credentials.

[sgales]

Yeah agents are running, is it possible the passwords are getting interpolated in a funny way? Should they be wrapped in single quotes if they have special characters in bash?

# pmm-admin --server-url=https://<admin>:<pass>@localhost --server-insecure-tls status
Agent ID: pmm-server
Node ID : pmm-server

PMM Server:
	URL    : https://127.0.0.1:443/
	Version: 2.25.0

PMM Client:
	Connected        : true
	Time drift       : 58.388µs
	Latency          : 190.567µs
	pmm-admin version: 2.25.0
	pmm-agent version: 2.25.0
Agents:
	/agent_id/183869bc-9d52-41cd-bd2f-524a65d3550b mysqld_exporter Running
	/agent_id/2307a0ee-ea2e-4afe-9a71-d8b15a2e3030 mysqld_exporter Running
	/agent_id/2c4ddba4-68d3-481d-947e-54a7a9b03e36 mysql_perfschema_agent Running
	/agent_id/329dcc46-5f68-4bcb-9494-cecd84484e33 mysql_perfschema_agent Waiting
	/agent_id/3467b929-7689-4d6b-917e-82118a322605 mysql_perfschema_agent Waiting
	/agent_id/44b5d6b6-f555-42d2-8820-dccbcb1c7b95 node_exporter Running
	/agent_id/47778b1d-2f60-4fd6-aed9-5dbb2c42f4b9 mongodb_profiler_agent Running
	/agent_id/4e0b9e27-0802-4857-8920-aaa71ca2fdc1 mysql_perfschema_agent Running
	/agent_id/51744b84-093f-47c6-bbf6-8a0389979672 mysqld_exporter Running
	/agent_id/57073381-4b64-4cc5-98be-5b136b3b305f mysql_perfschema_agent Running
	/agent_id/5ee6b55e-746c-499a-aeaf-af67834f316f mysqld_exporter Running
	/agent_id/7deac61a-edfa-4fa3-a595-81a04b54d8df postgresql_pgstatements_agent Running
	/agent_id/80d8b91b-0fd2-450b-9d03-9d574328e8f6 mysql_perfschema_agent Running
	/agent_id/a163ba2b-a96f-470c-915d-4aa1f84966c6 mongodb_exporter Running
	/agent_id/bb7e49c3-6b19-47ce-855a-21d8d54acdd3 postgres_exporter Running
	/agent_id/bf1d7655-6085-465c-a6a0-4afac50f6d9c mongodb_profiler_agent Running
	/agent_id/c16465f0-7aad-4a06-99aa-3d33adc1ac21 mysqld_exporter Running
	/agent_id/eb4bffcf-35aa-44e2-bc33-c2adc05b38f8 mysqld_exporter Running
	/agent_id/febfa611-d518-4336-9098-3d93d681f4a0 mongodb_exporter Running
	pmm-server/rds rds_exporter Running

I am guessing I could do something like this:

pmm-agent --config-file=<path to config> --server_username=<pmm user> --server_password=<password> --server_address=<mongohost> ----listen-port=21337 --debug

Also tried via the mongoshell to connect to the host using <pmm_mongodb_user> / <pmm_mongodb_pass>

MongoDB Enterprise mongo_test:PRIMARY> use <profileddb>
switched to db <profileddb>
MongoDB Enterprise mongo_test:PRIMARY> show profile

query	<profileddb>.<db> 24ms Fri Feb 04 2022 16:19:11
command:{
	"find" : "<db>",
	"filter" : {

	},
	"lsid" : {
		"id" : UUID("<uuid>")
	},
	"$clusterTime" : {
		"clusterTime" : Timestamp(1644016738, 1),
		"signature" : {
			"hash" : BinData(0,"<data>"),
			"keyId" : NumberLong("<key>")
		}
	},

	....

Also seeing this come up from time to time:

WARN[2022-02-09T20:27:19.523+00:00] couldn't retrieve data from cursor connection() error occured during connection handshake: dial tcp: i/o timeout  agentID=/agent_id/47778b1d-2f60-4fd6-aed9-5dbb2c42f4b9 component=agent-builtin db=<dbmonitored> type=qan_mongodb_profiler_agent

[sgales]

@Ivan_Groenewold

Thanks for your help, well QAN metrics started to show up, we’re not entirely sure why, other than either the profiler wasn’t picking up the metrics we thought it should, or there was some networking stuff invisibly busted.

Either way I did have a followup question. It looks like the configuration is not setup so it can do explains on the actual query. Is this intentional? If I try to run an explain it will fail saying it doesn’t have reads enabled on the db and thus fails. We understand that just curious if that was the intention in the documentation as I don’t see anything that says to expect this behavior.

[Ivan_Groenewold]

Glad to hear it is working now. Does the explain not work even for find() statements? I seem to recall running into cases where explain does not work for statements that write (insert, update) unless the mongodb user is given the perms to actually execute the write operation even if it does not need it theoretically.
Also I assume you double-checked the required perms. Check this one to make sure you have properly set the custom explainRole:

Configuring a MongoDB Sharded Cluster with PMM2 - Part 2 - Percona Database...

[melvin]

Hi,

I might be facing a similar issue.

The mongodb server has pmm installed (client, server, etc.) and the dashboard is seen in the browser.
The “MongoDB Instances Overview” dashboard shows the Node and Service names correctly:

But, the “PMM Query Analytics” dashboard does Not show the node and the service names:

I believe, i missed a step. Unable to figure it out.
Please help.

• Melvin

[Ivan_Groenewold]

Hi Melvin, do you see anything on the system.profile collection? it could be simply that no slow queries are present.

[melvin]

Hi @Ivan_Groenewold , thanks for your response.

This issue is fixed.

Seems if i enabling profiling in mongo shell prompt [ db.setProfilingLevel(2, {“slowms”: 0, “sampleRate”: 1.0}) ], the node does Not get listed in QAN. But, if i permanently add the same profiling levels in /etc/mongod.conf file, and restart mongod service, then the node gets listed in QAN.

@Anil_Joshi from Percona support team helped me with this.

[Ayushi-gupta]

Hy @melvin Could you please tell me which steps you followed to get this error done, I am also facing the same issue and cannot figure it out.

[Ayushi-gupta]

where we can get the system.profile section? On the PMM dashboard itself?