PMM's MongoDB dashboards for "Index Scan Ratios" and "Document Scan Ratios" stay on "NO DATA"

Percona Monitoring and Management (PMM) PMM 2.x
1

Description:

I just configured my MongoDB 6.0.6 instance to register at my PMM 2.40.1 server. That all worked out fine. I receive OS information and such.

Next, I created the two user roles as described in MongoDB - Percona Monitoring and Management. And then created the “pmm” user as described by the referenced document. I also made sure that the explainRole contained access to the system.profile collection for each DB as well, as prescribed by MongoDB - Percona Monitoring and Management.

In the MongoDB shell, I enabled profiling for the two databases I wanted to specifically monitor performance for, namely ixplatform_a and ixplatform_b, using this:

use ixplatform_a;
db.setProfilingLevel(2, {slowms: 100, sampleRate: 0.1});
use ixplatform_b;
db.setProfilingLevel(2, {slowms: 100, sampleRate: 0.1});

Next, I added / registered the MongoDB service to PMM, using this command:

pmm-admin add mongodb --query-source=profiler --username=pmm --password=xyz --cluster=cluster-name --enable-all-collectors --stats-collections=ixplatform_a,ixplatform_b server-name 127.0.0.1

Now, all Grafana dashboard fill up nicely and also the QAN receives query usage information. However, the only two Grafana graphs / dashboards that stay empty, are the ones under “Query Efficiency Detail”:

  • Index Scan Ratios
  • Document Scan Ratios

What am I missing here? Is the pmm-admin add mongodb command wrong? Are the permissions for the MongoDB “pmm” user wrong? Should I give more specific access to the ixplatform_a & ixplatform_b databases?

Version:

MongoDB: 6.0.6
PMM: 2.40.1

2

To add to this, it looks like the mongodb_exporter gets the correct calling arguments:

$ ps ax | grep mongodb_exporter
1694059 ?        Sl    93:44 /usr/local/percona/pmm2/exporters/mongodb_exporter --collector.collstats --collector.collstats-limit=200 --collector.dbstats --collector.diagnosticdata --collector.indexstats --collector.replicasetstatus --collector.topmetrics --compatible-mode --discovering-mode --log.level=warn --mongodb.collstats-colls=ixplatform_a,ixplatform_b --mongodb.global-conn-pool --mongodb.indexstats-colls=ixplatform_a,ixplatform_b --web.listen-address=:42000 --web.config=/usr/local/percona/pmm2/tmp/mongodb_exporter/agent_id/11...8a/webConfigPlaceholder

When I login in MongoDB as the “pmm” MongoDB user, I can also execute some of the queries I find in the exporter’s source (https://github.com/percona/mongodb_exporter/blob/main/exporter/indexstats_collector.go#L90-L94):

cluster_repl_set_1 [direct: primary] test> use ixplatform_a
switched to db ixplatform_a
cluster_repl_set_1 [direct: primary] ixplatform_logs> db.collection_with_indexes.aggregate({$indexStats: {}})
[
  {
    name: 'client_public_id_1_timestamp_-1',
    key: { client_public_id: 1, timestamp: -1 },
    host: '...',
    accesses: {
      ops: Long("2215973425"),
      since: ISODate("2023-08-15T07:05:53.067Z")
    },
    spec: {
      v: 2,
      key: { client_public_id: 1, timestamp: -1 },
      name: 'client_public_id_1_timestamp_-1'
    }
  },
  {
    name: '_id_',
    key: { _id: 1 },
    host: '...',
    accesses: { ops: Long("0"), since: ISODate("2023-08-15T07:05:53.067Z") },
    spec: { v: 1, key: { _id: 1 }, name: '_id_' }
  }
]

What else could be wrong or is worth checking out?

3

Hi @hvtilborg , how many collections do you have? As far as I see you might not get these metrics in case you have more then 200 collections. We implemented that because it runs the query for each collection and might affect performance. If it’s the case you can re-add your service with --max-collections-limit=0 and the limit will be disabled.

4

Hey @nurlan , not that many:

cluster_repl_set_1 [direct: secondary] ixplatform_a> show collections
(...3 collections...)
cluster_repl_set_1 [direct: secondary] ixplatform_b> show collections
(...11 collections...)

I did however re-add the mongodb service anyway, using:

pmm-admin add mongodb --query-source=profiler --username=pmm --password=xyz --cluster=cluster-name --enable-all-collectors --stats-collections=ixplatform_a,ixplatform_b --max-collections-limit=0 -- server-name 127.0.0.1

And I see this reflected in the mongodb_exporter command arguments using ps:

 826197 ?        Sl     2:05 /usr/local/percona/pmm2/exporters/mongodb_exporter --collector.collstats --collector.collstats-limit=0 --collector.dbstats --collector.diagnosticdata --collector.indexstats --collector.replicasetstatus --collector.topmetrics --compatible-mode --discovering-mode --log.level=warn --mongodb.collstats-colls=ixplatform_a,ixplatform_b --mongodb.global-conn-pool --mongodb.indexstats-colls=ixplatform_a,ixplatform_b --web.listen-address=:42003 --web.config=/usr/local/percona/pmm2/tmp/mongodb_exporter/agent_id/c2...71/webConfigPlaceholder

However, the two dashboards stay empty :frowning: :

image
image1065×641 34.9 KB

5

Does anyone have a hint what causes these four missing graphs?

6

Hi @hvtilborg,
Could you run pmm-admin summary and check logs for mongodb_exporter if there are anything in logs?
Is it a cluster? if it is, which instances have you added?

7

Hi @nurlan,

It is a cluster, not a sharded one but containing a replicaset. The cluster consists of three nodes – am04, am05, and am07 – where am05 is currently the primary. The pmm-agent is installed on all three nodes.

When I run pmm-admin summary on the am05, I do get this error:

[GET /logs.zip][401] Logs default  &{Code:7 Error:Access denied. Message:Access denied.}
summary_am05_2023_11_16_09_33_46.zip created.

But the ZIP does contain the log files of the mongodb_exporter in client/pmm-agent/MONGODB_EXPORTER e03e1a6f-4d1b-4298-8411-3950532f5774.log. Here a small summary of that:

time="2023-11-07T10:20:55.456+00:00" level=error msg="time=\"2023-11-07T10:20:55Z\" level=error msg=\"Registry - Cannot get arbiterOnly to check if this is arbiter role : cannot check if the instance is an arbiter: server selection error: context canceled, current topology: { Type: Single, Servers: [{ Addr: 127.0.0.1:6317, Type: Unknown, Last error: (ShutdownInProgress) The server is in quiesce mode and will shut down }, ] }\"" agentID=/agent_id/e03e1a6f-4d1b-4298-8411-3950532f5774 component=agent-process type=mongodb_exporter
time="2023-11-07T10:20:55.954+00:00" level=error msg="time=\"2023-11-07T10:20:55Z\" level=error msg=\"Registry - Cannot get node type to check if this is a mongos : server selection error: context canceled, current topology: { Type: Single, Servers: [{ Addr: 127.0.0.1:6317, Type: Unknown, Last error: (ShutdownInProgress) The server is in quiesce mode and will shut down }, ] }\"" agentID=/agent_id/e03e1a6f-4d1b-4298-8411-3950532f5774 component=agent-process type=mongodb_exporter
time="2023-11-07T10:20:55.954+00:00" level=error msg="time=\"2023-11-07T10:20:55Z\" level=error msg=\"Registry - Cannot get arbiterOnly to check if this is arbiter role : cannot check if the instance is an arbiter: server selection error: context canceled, current topology: { Type: Single, Servers: [{ Addr: 127.0.0.1:6317, Type: Unknown, Last error: (ShutdownInProgress) The server is in quiesce mode and will shut down }, ] }\"" agentID=/agent_id/e03e1a6f-4d1b-4298-8411-3950532f5774 component=agent-process type=mongodb_exporter
time="2023-11-07T10:20:59.457+00:00" level=warning msg="time=\"2023-11-07T10:20:59Z\" level=warning msg=\"cannot load topology labels: cannot check if the instance is an arbiter: server selection error: server selection timeout, current topology: { Type: Single, Servers: [{ Addr: 127.0.0.1:6317, Type: Unknown, Last error: connection(127.0.0.1:6317[-75]) incomplete read of message header: read tcp 127.0.0.1:50688->127.0.0.1:6317: i/o timeout }, ] }\"" agentID=/agent_id/e03e1a6f-4d1b-4298-8411-3950532f5774 component=agent-process type=mongodb_exporter
time="2023-11-07T10:21:00.458+00:00" level=error msg="time=\"2023-11-07T10:21:00Z\" level=error msg=\"error while checking mongodb connection: server selection error: server selection timeout, current topology: { Type: Single, Servers: [{ Addr: 127.0.0.1:6317, Type: Unknown, Last error: dial tcp 127.0.0.1:6317: connect: connection refused }, ] }. mongo_up is set to 0\"" agentID=/agent_id/e03e1a6f-4d1b-4298-8411-3950532f5774 component=agent-process type=mongodb_exporter
time="2023-11-07T10:21:00.955+00:00" level=error msg="time=\"2023-11-07T10:21:00Z\" level=error msg=\"Registry - Cannot get node type to check if this is a mongos : server selection error: context canceled, current topology: { Type: Single, Servers: [{ Addr: 127.0.0.1:6317, Type: Unknown, Last error: dial tcp 127.0.0.1:6317: connect: connection refused }, ] }\"" agentID=/agent_id/e03e1a6f-4d1b-4298-8411-3950532f5774 component=agent-process type=mongodb_exporter
time="2023-11-07T10:21:00.955+00:00" level=error msg="time=\"2023-11-07T10:21:00Z\" level=error msg=\"Registry - Cannot get arbiterOnly to check if this is arbiter role : cannot check if the instance is an arbiter: server selection error: context canceled, current topology: { Type: Single, Servers: [{ Addr: 127.0.0.1:6317, Type: Unknown, Last error: dial tcp 127.0.0.1:6317: connect: connection refused }, ] }\"" agentID=/agent_id/e03e1a6f-4d1b-4298-8411-3950532f5774 component=agent-process type=mongodb_exporter
time="2023-11-07T10:21:01.571+00:00" level=error msg="time=\"2023-11-07T10:21:01Z\" level=error msg=\"cannot retrieve engine type: Engine is unavailable\"" agentID=/agent_id/e03e1a6f-4d1b-4298-8411-3950532f5774 component=agent-process type=mongodb_exporter

But looking at the date of the logs, this was around when we installed PMM on this specific node. (We interchanged two of the three nodes in the meanwhile of this topic.) There are no current mongodb_exporter logs.

8

I was wondering if anyone has another idea what is going wrong here.