pmm2 client setup in Sharded MongoDB

It’s not really clear how to set up mongodb_exporter user in case of Sharded MongoDB cluster. There are Sharded Cluster Users and Shard Local Users -…. Where a user for mongodb_exporter   should  be created?  Should it be created as a “sharded cluster user” on mongos and not as a shard local directly on mongod node?
If so, should the QAN agent be running on mongos and not on mongod nodes in this case?
Thank you in advance for your help!

Hi Eugene,
It’s required to use exporter on mongos in order to collect sharding metrics (mongodb_mongos_sharding_*).

Thanks for answer. It’s still not really clear to me.
Let’s say I have two Replica Sets in Sharded cluster setup. I have 3 servers in replica set rs0, let’s name them mongod-rs0-1, mongod-rs0-2, mongod-rs0-3.  And another 3 server in replica set rs1 - mongod-rs1-1, mongod-rs1-2, mongod-rs1-3. And yet another 3 server in configuration replica set mongod-cfg-1, mongod-cfg-2, mongod-cfg-3. And there are 2 server running mongoses - mongos-1, mongos-2.
So, there are a total of 11 nodes in the cluster.

Now, how do I configure PMM2 clients for that sharded cluster?
1. At first, install pmm2-client on all 11 nodes and run “pmm-admin config --server-insecure-tls --server-url=<pmm-server>”. At this point we have node_exporter running on every node.
2. Then I should add mongodb monitoring:
  a) According to the docs, “MongoDB should be prepared for the monitoring”, i.e. profiling should be enabled and mongodb_exporter user has to be setup. What I understood from your answer is that mongodb_exporter user needs to be set up once on mongos instance as Sharded Cluster User. It should not be created on every mongod node which is part of replica sets. Is that right?
  b) After that, where should I add mongodb service “pmm-admin add mongodb”? On mongos nodes only? Or still on every other node in the cluster? Here comes the confusion. It does not seem right that mongodb monitoring is added on mongos nodes only and not on all other nodes of the cluster. Yet, mongod replica set nodes do not have mongodb permissions setup and it’s not clear if running mongodb_exporter and mongodb_profiler_agent on these nodes makes sense.

Please clarify the proper setup of PMM2 on sharded clusters. I see that the PMM documentation is lacking on this matters.
Thank you. 

@adivinho Could you please reply to the above? Thank you for your help

Hi Eugen,
MongoDB service monitoring should be added on all nodes. So please create mongodb_exporter user on each PRIMARY node and use it for connecting nodes in replica sets.

Here are commands of enabling monitoring for nodes in the same replica set “rs1”.
# pmm-admin add mongodb --username=mongodb_exporter --password=mongo --replication-set=rs1 --environment=mongodb_rs_node mongodb_rs1_1 pmm-admin add mongodb --username=mongodb_exporter --password=mongo --replication-set=rs1 --environment=mongodb_rs_node mongodb_rs1_2 pmm-admin add mongodb --username=mongodb_exporter --password=mongo --replication-set=rs1 --environment=mongodb_rs_node mongodb_rs1_3

So, it means to create mongodb_exporter user as Sharded Cluster User on mongos nodes to collect sharding metrics (mongodb_mongos_sharding_*) and to create mongodb_exporter user as Shard Local User on primary nodes to monitor mongod replica set member nodes. Is this correct?

Yes. It’s correct.