# Metrics are not getting populated to PMM-server after successfully configuring the client with the Server **URL:** \ **Category:** Uncategorized\ **Tags:** pmm, percona\ **Created:** [April 14, 2022, 10:26pm UTC](https://forums.percona.com/t/metrics-are-not-getting-populated-to-pmm-server-after-successfully-configuring-the-client-with-the-server/15241 "2022-04-14T22:26:59Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![Karthikyvpv](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/karthikyvpv/32/13781_2.png) [@Karthikyvpv](https://forums.percona.com/u/Karthikyvpv)\ **Post date:** [April 14, 2022, 10:26pm UTC](https://forums.percona.com/t/metrics-are-not-getting-populated-to-pmm-server-after-successfully-configuring-the-client-with-the-server/15241/1 "2022-04-14T22:26:59Z")
After running the commnad below, the pmm agent starts successfully but its not able to populate the metrics data. I also see the node getting registered on the PMM server as well as its visible in the Grafana dashboard. But the metrics data does not populate. Below is the error from the syslog: ```auto INFO[2022-04-14T15:25:35.092-07:00] 2022-04-14T22:25:35.092Z error VictoriaMetrics/lib/promscrape/scrapework.go:354error when scraping "http://127.0.0.1:42000/metrics?collect%5B%5D=buddyinfo&collect%5B%5D=cpu&collect%5B%5D=diskstats&collect%5B%5D=filefd&collect%5B%5D=filesystem&collect%5B%5D=loadavg&collect%5B%5D=meminfo&collect%5B%5D=meminfo_numa&collect%5B%5D=netdev&collect%5B%5D=netstat&collect%5B%5D=processes&collect%5B%5D=standard.go&collect%5B%5D=standard.process&collect%5B%5D=stat&collect%5B%5D=textfile.hr&collect%5B%5D=time&collect%5B%5D=vmstat" from job "node_exporter_agent_id_215daabc-96e7-4bdd-aa12-26a826c92c9e_hr-1s" with labels {agent_id="/agent_id/215daabc-96e7-4bdd-aa12-26a826c92c9e",agent_type="node_exporter",instance="/agent_id/215daabc-96e7-4bdd-aa12-26a826c92c9e",job="node_exporter_agent_id_215daabc-96e7-4bdd-aa12-26a826c92c9e_hr-1s",machine_id="/machine_id/cd7ecc10985f4f65a08f0ba525114dc0",node_id="/node_id/f09e4234-5f04-40e7-9a8f-c082d3851d08",node_name="ht-prod-tcdb2.spacex.corp",node_type="generic"}: error when scraping "http://127.0.0.1:42000/metrics?collect%5B%5D=buddyinfo&collect%5B%5D=cpu&collect%5B%5D=diskstats&collect%5B%5D=filefd&collect%5B%5D=filesystem&collect%5B%5D=loadavg&collect%5B%5D=meminfo&collect%5B%5D=meminfo_numa&collect%5B%5D=netdev&collect%5B%5D=netstat&collect%5B%5D=processes&collect%5B%5D=standard.go&collect%5B%5D=standard.process&collect%5B%5D=stat&collect%5B%5D=textfile.hr&collect%5B%5D=time&collect%5B%5D=vmstat" with timeout 1s: timeout agentID=/agent_id/e5db1093-68ec-4623-992f-67e9ccf4daed component=agent-process type=vm_agent ``` ---
**Author:** ![matthewb](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/matthewb/32/34_2.png) [@matthewb](https://forums.percona.com/u/matthewb)\ **Post date:** [April 15, 2022, 2:42pm UTC](https://forums.percona.com/t/metrics-are-not-getting-populated-to-pmm-server-after-successfully-configuring-the-client-with-the-server/15241/2 "2022-04-15T14:42:27Z")
There is not much information there. Can you please show how you set up PMM, how did you register the agent to the server, and how did you add mysql service to PMM? ---
**Author:** ![Karthikyvpv](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/karthikyvpv/32/13781_2.png) [@Karthikyvpv](https://forums.percona.com/u/Karthikyvpv)\ **Post date:** [April 15, 2022, 8:29pm UTC](https://forums.percona.com/t/metrics-are-not-getting-populated-to-pmm-server-after-successfully-configuring-the-client-with-the-server/15241/3 "2022-04-15T20:29:40Z")
sudo yum install [https://repo.percona.com/yum/percona-release-latest.noarch.rpm](https://repo.percona.com/yum/percona-release-latest.noarch.rpm) Install the PMM Client package – yum install -y pmm2-client Register PMM Server – pmm-admin config --server-url=‘[https://admin:\*\*\*\*\*\*\*\*@ht1-perc-mntr01:443](https://admin: ******** @ht1-perc-mntr01:443)’ --server-insecure-tls I used the above commands to set up PMM client. I just registered the agent with the server. I have not yet added Mysql service. ---
**Author:** ![matthewb](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/matthewb/32/34_2.png) [@matthewb](https://forums.percona.com/u/matthewb)\ **Post date:** [April 15, 2022, 9:48pm UTC](https://forums.percona.com/t/metrics-are-not-getting-populated-to-pmm-server-after-successfully-configuring-the-client-with-the-server/15241/4 "2022-04-15T21:48:20Z")
`pmm-admin status` `pmm-admin list` `ps -Af | grep pmm` Can you do some basic sysadmin checking on if things are even running? ---
**Author:** ![Karthikyvpv](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/karthikyvpv/32/13781_2.png) [@Karthikyvpv](https://forums.percona.com/u/Karthikyvpv)\ **Post date:** [April 18, 2022, 6:12pm UTC](https://forums.percona.com/t/metrics-are-not-getting-populated-to-pmm-server-after-successfully-configuring-the-client-with-the-server/15241/5 "2022-04-18T18:12:04Z")
```auto [root@ht-prod-tcdb2 ~]# pmm-admin status Agent ID: /agent_id/b01c0ffd-0cbb-46c9-9f03-d63d45ec2c78 Node ID : /node_id/f09e4234-5f04-40e7-9a8f-c082d3851d08 PMM Server: URL : https://ht1-perc-mntr01:443/ Version: 2.15.1 PMM Client: Connected : true Time drift : 102.469µs Latency : 818.838µs pmm-admin version: 2.27.0 pmm-agent version: 2.27.0 Agents: /agent_id/215daabc-96e7-4bdd-aa12-26a826c92c9e node_exporter Running /agent_id/e5db1093-68ec-4623-992f-67e9ccf4daed vmagent Running [root@ht-prod-tcdb2 ~]# pmm-admin list Service type Service name Address and port Service ID Agent type Status Metrics Mode Agent ID Service ID pmm_agent Connected /agent_id/b01c0ffd-0cbb-46c9-9f03-d63d45ec2c78 node_exporter Running push /agent_id/215daabc-96e7-4bdd-aa12-26a826c92c9e vmagent Running push /agent_id/e5db1093-68ec-4623-992f-67e9ccf4daed [root@ht-prod-tcdb2 ~]# ps -Af | grep pmm root 2430 2058 0 11:11 pts/0 00:00:00 grep --color=auto pmm root 3786 1 0 Apr14 ? 00:01:33 /usr/sbin/pmm-agent --config-file=/usr/local/percona/pmm2/config/pmm-agent.yaml root 4273 3786 27 Apr14 ? 1-01:34:54 /usr/local/percona/pmm2/exporters/node_exporter --collector.bonding --collector.buddyinfo --collector.cpu --collector.diskstats --collector.entropy --collector.filefd --collector.filesystem --collector.hwmon --collector.loadavg --collector.meminfo --collector.meminfo_numa --collector.netdev --collector.netstat --collector.netstat.fields=^(.*_(InErrors|InErrs|InCsumErrors)|Tcp_(ActiveOpens|PassiveOpens|RetransSegs|CurrEstab|AttemptFails|OutSegs|InSegs|EstabResets|OutRsts|OutSegs)|Tcp_Rto(Algorithm|Min|Max)|Udp_(RcvbufErrors|SndbufErrors)|Udp(6?|Lite6?)_(InDatagrams|OutDatagrams|RcvbufErrors|SndbufErrors|NoPorts)|Icmp6?_(OutEchoReps|OutEchos|InEchos|InEchoReps|InAddrMaskReps|InAddrMasks|OutAddrMaskReps|OutAddrMasks|InTimestampReps|InTimestamps|OutTimestampReps|OutTimestamps|OutErrors|InDestUnreachs|OutDestUnreachs|InTimeExcds|InRedirects|OutRedirects|InMsgs|OutMsgs)|IcmpMsg_(InType3|OutType3)|Ip(6|Ext)_(InOctets|OutOctets)|Ip_Forwarding|TcpExt_(Listen.*|Syncookies.*|TCPTimeouts))$ --collector.processes --collector.standard.go --collector.standard.process --collector.stat --collector.textfile.directory.hr=/usr/local/percona/pmm2/collectors/textfile-collector/high-resolution --collector.textfile.directory.lr=/usr/local/percona/pmm2/collectors/textfile-collector/low-resolution --collector.textfile.directory.mr=/usr/local/percona/pmm2/collectors/textfile-collector/medium-resolution --collector.textfile.hr --collector.textfile.lr --collector.textfile.mr --collector.time --collector.uname --collector.vmstat --collector.vmstat.fields=^(pg(steal_(kswapd|direct)|refill|alloc)_(movable|normal|dma3?2?)|nr_(dirty.*|slab.*|vmscan.*|isolated.*|free.*|shmem.*|i?n?active.*|anon_transparent_.*|writeback.*|unstable|unevictable|mlock|mapped|bounce|page_table_pages|kernel_stack)|drop_slab|slabs_scanned|pgd?e?activate|pgpg(in|out)|pswp(in|out)|pgm?a?j?fault)$ --no-collector.arp --no-collector.bcache --no-collector.conntrack --no-collector.drbd --no-collector.edac --no-collector.infiniband --no-collector.interrupts --no-collector.ipvs --no-collector.ksmd --no-collector.logind --no-collector.mdadm --no-collector.mountstats --no-collector.netclass --no-collector.nfs --no-collector.nfsd --no-collector.ntp --no-collector.qdisc --no-collector.runit --no-collector.sockstat --no-collector.supervisord --no-collector.systemd --no-collector.tcpstat --no-collector.timex --no-collector.wifi --no-collector.xfs --no-collector.zfs --web.disable-exporter-metrics --web.listen-address=:42000 root 4345 3786 0 Apr14 ? 00:17:22 /usr/local/percona/pmm2/exporters/vmagent -envflag.enable=true -httpListenAddr=127.0.0.1:42001 -loggerLevel=INFO -promscrape.config=/tmp/vm_agent/agent_id/e5db1093-68ec-4623-992f-67e9ccf4daed/vmagentscrapecfg -remoteWrite.maxDiskUsagePerURL=1073741824 -remoteWrite.tlsInsecureSkipVerify=true -remoteWrite.tmpDataPath=/tmp/vmagent-temp-dir -remoteWrite.url=https://ht1-perc-mntr01:443/victoriametrics/api/v1/write [root@ht-prod-tcdb2 ~]# ``` ---
**Author:** ![matthewb](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/matthewb/32/34_2.png) [@matthewb](https://forums.percona.com/u/matthewb)\ **Post date:** [April 18, 2022, 6:46pm UTC](https://forums.percona.com/t/metrics-are-not-getting-populated-to-pmm-server-after-successfully-configuring-the-client-with-the-server/15241/6 "2022-04-18T18:46:46Z")
Everything looks correct and appears to be running. But your CPU graphs are empty? ---
**Author:** ![Karthikyvpv](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/karthikyvpv/32/13781_2.png) [@Karthikyvpv](https://forums.percona.com/u/Karthikyvpv)\ **Post date:** [April 18, 2022, 10:22pm UTC](https://forums.percona.com/t/metrics-are-not-getting-populated-to-pmm-server-after-successfully-configuring-the-client-with-the-server/15241/7 "2022-04-18T22:22:33Z")
Yes. There is no graph getting populated. And I shared the error messages from /var/log/messages on the top ---
**Author:** ![matthewb](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/matthewb/32/34_2.png) [@matthewb](https://forums.percona.com/u/matthewb)\ **Post date:** [April 19, 2022, 3:23pm UTC](https://forums.percona.com/t/metrics-are-not-getting-populated-to-pmm-server-after-successfully-configuring-the-client-with-the-server/15241/8 "2022-04-19T15:23:35Z")
Can you run `ss -nltp | grep 42000` and see if PMM is listening on that port? If so, you probably have some firewall which is blocking PMM from scraping its own data. ---
**Author:** ![Karthikyvpv](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/karthikyvpv/32/13781_2.png) [@Karthikyvpv](https://forums.percona.com/u/Karthikyvpv)\ **Post date:** [April 19, 2022, 7:53pm UTC](https://forums.percona.com/t/metrics-are-not-getting-populated-to-pmm-server-after-successfully-configuring-the-client-with-the-server/15241/9 "2022-04-19T19:53:22Z")
> [@matthewb](#): > > ss -nltp | grep 42000 ```auto ss -nltp | grep 42000 LISTEN 0 128 [::]:42000 [::]:* users:(("node_exporter",pid=4273,fd=3)) `Preformatted text``Preformatted text` ``` its listening. ---
**Author:** ![matthewb](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/matthewb/32/34_2.png) [@matthewb](https://forums.percona.com/u/matthewb)\ **Post date:** [April 19, 2022, 8:23pm UTC](https://forums.percona.com/t/metrics-are-not-getting-populated-to-pmm-server-after-successfully-configuring-the-client-with-the-server/15241/10 "2022-04-19T20:23:57Z")
Does this return anything? `curl http://127.0.0.1:42000/metrics?collect%5B%5D=buddyinfo&collect%5B%5D=cpu` If not, then you have firewall rules blocking localhost. ---
**Author:** ![Karthikyvpv](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/karthikyvpv/32/13781_2.png) [@Karthikyvpv](https://forums.percona.com/u/Karthikyvpv)\ **Post date:** [April 20, 2022, 5:49pm UTC](https://forums.percona.com/t/metrics-are-not-getting-populated-to-pmm-server-after-successfully-configuring-the-client-with-the-server/15241/11 "2022-04-20T17:49:39Z")
It says Invalid Username or password… ```auto curl http://127.0.0.1:42000/metrics?collect Invalid username or password ``` ---
**Author:** ![Karthikyvpv](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/karthikyvpv/32/13781_2.png) [@Karthikyvpv](https://forums.percona.com/u/Karthikyvpv)\ **Post date:** [May 13, 2022, 5:57pm UTC](https://forums.percona.com/t/metrics-are-not-getting-populated-to-pmm-server-after-successfully-configuring-the-client-with-the-server/15241/12 "2022-05-13T17:57:39Z")
Hello, I identified the issue. The node\_exported is timing out because there are so too many virtual filesystems and the exporter is timing out while collecting those metrics. Command I ran for that - lsof -p $(pidof node\_exporter) i can see all the virtual filesystems… and a simple df -k takes more than 5s to respond. Is there any way to fix the timeout parameter for the exporter?