Several metrics missing after adding service

Percona Monitoring and Management (PMM) PMM 2.x
,
1

PMM Server Version: 2.19.0
pmm-admin version: 2.23.0
pmm-agent version: 2.23.0

I have one EC2 instance running MySQL 8, where I have installed pmm-client using apt-get, and one PMM server (installed using docker) running behind AWS ALB.

  1. When I connect client to server like this pmm-admin config --server-insecure-tls --server-url=[https://admin:<password>@pmm.example.com](https://admin:%3Cpassword%3E@pmm.example.com) it shows that pmm-client is running but I don’t see the node in the home dashboard of PMM server. Though I can see the node in inventory with the type as “Generic”.

  2. I am unable to add service from DB host. I run pmm-admin add mysql --username=pmm --password=<password> --query-source=perfschema name but it gives no output on terminal.

  3. I can add MySQL instance/service from the UI. When I do this it creates a service as well as a node with type “Remote” in the inventory. I can see this node in the home dashboard of PMM server but it doesn’t show CPU, disk, memory, network metrics.

I am running PMM server behind AWS ALB.

I followed Network - Percona Monitoring and Management for networking rules. I have following rules open:

  1. Port 3306 on client from server
  2. Port 42000-51999 on client from server
  3. Port 8428 on client from server
  4. Port 7771 on server from client
  5. Port 443 on server from client
  6. All outbound rules are open in client and server

I only see below logs in the client (/var/log/syslog):

Nov 18 07:53:20 eta-c pmm-agent[119616]: #033[36mINFO#033[0m[2021-11-18T07:53:20.668+00:00] 2021-11-18T07:53:20.668Z#011error#011VictoriaMetrics/app/vmagent/remotewrite/client.go:308#011unexpected status code received after sending a block with size 24586 bytes to "1:secret-url" during retry #83: 464; response body=""; re-sending the block in 60.000 seconds #033[36magentID#033[0m=/agent_id/f80a3901-0670-4ac7-97ab-b06ee4010f45 #033[36mcomponent#033[0m=agent-process #033[36mtype#033[0m=vm_agent

Nov 18 07:53:20 eta-c pmm-agent[119616]: #033[36mINFO#033[0m[2021-11-18T07:53:20.669+00:00] 2021-11-18T07:53:20.669Z#011error#011VictoriaMetrics/app/vmagent/remotewrite/client.go:308#011unexpected status code received after sending a block with size 944 bytes to "1:secret-url" during retry #84: 464; response body=""; re-sending the block in 60.000 seconds #033[36magentID#033[0m=/agent_id/f80a3901-0670-4ac7-97ab-b06ee4010f45 #033[36mcomponent#033[0m=agent-process #033[36mtype#033[0m=vm_agent

Nov 18 07:53:20 eta-c pmm-agent[119616]: #033[36mINFO#033[0m[2021-11-18T07:53:20.669+00:00] 2021-11-18T07:53:20.669Z#011error#011VictoriaMetrics/app/vmagent/remotewrite/client.go:308#011unexpected status code received after sending a block with size 1344 bytes to "1:secret-url" during retry #84: 464; response body=""; re-sending the block in 60.000 seconds #033[36magentID#033[0m=/agent_id/f80a3901-0670-4ac7-97ab-b06ee4010f45 #033[36mcomponent#033[0m=agent-process #033[36mtype#033[0m=vm_agent

Nov 18 07:53:20 eta-c pmm-agent[119616]: #033[36mINFO#033[0m[2021-11-18T07:53:20.669+00:00] 2021-11-18T07:53:20.669Z#011error#011VictoriaMetrics/app/vmagent/remotewrite/client.go:308#011unexpected status code received after sending a block with size 24605 bytes to "1:secret-url" during retry #83: 464; response body=""; re-sending the block in 60.000 seconds #033[36magentID#033[0m=/agent_id/f80a3901-0670-4ac7-97ab-b06ee4010f45 #033[36mcomponent#033[0m=agent-process #033[36mtype#033[0m=vm_agent

What am I missing?

2

Hi. Can you give some more information - logs from /srv/logs/pmm-agent.log, pmm-admin list and pmm-admin status.

1 Like