ProxySQL added to "PPM Inventory" but the status remain N/A

1

I’ve added a ProxySQL Service to the PMM Inventory by running

pmm-admin add proxysql --username=radmin --password=radmin --host=172.30.18.212 --port=6032 --service-name=proxysql-service --debug

on the ProxySQL instance

The command debug output is as follows:

ubuntu@ip-172-30-18-212:~$ pmm-admin add proxysql --username=radmin --password=radmin --host=172.30.18.212 --port=6032 --service-name=proxysql-service --debug
DEBUG 2025-04-15 07:25:10.008537764Z: POST /local/Status HTTP/1.1
Host: 127.0.0.1:7777
User-Agent: Go-http-client/1.1
Content-Length: 3
Accept: application/json
Content-Type: application/json
Accept-Encoding: gzip

{}

DEBUG 2025-04-15 07:25:10.009743076Z: HTTP/1.1 200 OK
Content-Length: 1138
Content-Type: application/json
Date: Tue, 15 Apr 2025 07:25:10 GMT
Grpc-Metadata-Content-Type: application/grpc

{
“agent_id”: “/agent_id/04b9c01f-e53a-4786-aaf4-5bbb2607a5d9”,
“runs_on_node_id”: “/node_id/5283e9c5-8545-4245-a942-6ba7899ad6d4”,
“server_info”: {
“url”: “https://api_key:eyJrIjoiZFc2Qkt5QzZFMzUyMk93VGtxbDVTVHJmcHcyN25wVUsiLCJuIjoicG1tLWFnZW50LWlwLTE3Mi0zMC0xOC0yMTItODM4ODkwMTkzNTk0NDU2Njk3OCIsImlkIjoxfQ==@172.30.18.50:443/”,
“insecure_tls”: true,
“connected”: true,
“version”: “2.44.0”,
“latency”: null,
“clock_drift”: null
},
“agents_info”: [
{
“agent_id”: “/agent_id/47f5aaea-eb7e-42ac-a145-efdded352a42”,
“agent_type”: “NODE_EXPORTER”,
“status”: “RUNNING”,
“listen_port”: 42000,
“process_exec_path”: “/usr/local/percona/pmm2/exporters/node_exporter”
},
{
“agent_id”: “/agent_id/c08d974a-65ae-4b5d-ad5d-7d3718968815”,
“agent_type”: “VM_AGENT”,
“status”: “RUNNING”,
“listen_port”: 42001,
“process_exec_path”: “/usr/local/percona/pmm2/exporters/vmagent”
}
],
“config_filepath”: “/usr/local/percona/pmm2/config/pmm-agent.yaml”,
“agent_version”: “2.44.0”,
“node_name”: “ip-172-30-18-212”,
“connection_uptime”: 100
}
DEBUG 2025-04-15 07:25:10.010142783Z: POST /local/Status HTTP/1.1
Host: 127.0.0.1:7777
User-Agent: Go-http-client/1.1
Content-Length: 3
Accept: application/json
Content-Type: application/json
Accept-Encoding: gzip

{}

DEBUG 2025-04-15 07:25:10.011135461Z: HTTP/1.1 200 OK
Content-Length: 1138
Content-Type: application/json
Date: Tue, 15 Apr 2025 07:25:10 GMT
Grpc-Metadata-Content-Type: application/grpc

{
“agent_id”: “/agent_id/04b9c01f-e53a-4786-aaf4-5bbb2607a5d9”,
“runs_on_node_id”: “/node_id/5283e9c5-8545-4245-a942-6ba7899ad6d4”,
“server_info”: {
“url”: “https://api_key:eyJrIjoiZFc2Qkt5QzZFMzUyMk93VGtxbDVTVHJmcHcyN25wVUsiLCJuIjoicG1tLWFnZW50LWlwLTE3Mi0zMC0xOC0yMTItODM4ODkwMTkzNTk0NDU2Njk3OCIsImlkIjoxfQ==@172.30.18.50:443/”,
“insecure_tls”: true,
“connected”: true,
“version”: “2.44.0”,
“latency”: null,
“clock_drift”: null
},
“agents_info”: [
{
“agent_id”: “/agent_id/47f5aaea-eb7e-42ac-a145-efdded352a42”,
“agent_type”: “NODE_EXPORTER”,
“status”: “RUNNING”,
“listen_port”: 42000,
“process_exec_path”: “/usr/local/percona/pmm2/exporters/node_exporter”
},
{
“agent_id”: “/agent_id/c08d974a-65ae-4b5d-ad5d-7d3718968815”,
“agent_type”: “VM_AGENT”,
“status”: “RUNNING”,
“listen_port”: 42001,
“process_exec_path”: “/usr/local/percona/pmm2/exporters/vmagent”
}
],
“config_filepath”: “/usr/local/percona/pmm2/config/pmm-agent.yaml”,
“agent_version”: “2.44.0”,
“node_name”: “ip-172-30-18-212”,
“connection_uptime”: 100
}
DEBUG 2025-04-15 07:25:10.011413976Z: POST /v1/management/ProxySQL/Add HTTP/1.1
Host: 172.30.18.50:443
User-Agent: Go-http-client/1.1
Content-Length: 303
Accept: application/json
Authorization: Basic YXBpX2tleTpleUpySWpvaVpGYzJRa3Q1UXpaRk16VXlNazkzVkd0eGJEVlRWSEptY0hjeU4yNXdWVXNpTENKdUlqb2ljRzF0TFdGblpXNTBMV2x3TFRFM01pMHpNQzB4T0MweU1USXRPRE00T0Rrd01Ua3pOVGswTkRVMk5qazNPQ0lzSW1sa0lqb3hmUT09
Content-Type: application/json
Accept-Encoding: gzip

{“node_id”:“/node_id/5283e9c5-8545-4245-a942-6ba7899ad6d4”,“service_name”:“proxysql-service”,“address”:“172.30.18.212”,“port”:6032,“pmm_agent_id”:“/agent_id/04b9c01f-e53a-4786-aaf4-5bbb2607a5d9”,“username”:“radmin”,“password”:“radmin”,“metrics_mode”:“AUTO”,“disable_collectors”:null,“log_level”:“warn”}

DEBUG 2025-04-15 07:25:10.042744393Z: HTTP/1.1 200 OK
Content-Length: 604
Cache-Control: no-cache
Connection: keep-alive
Content-Type: application/json
Date: Tue, 15 Apr 2025 07:25:10 GMT
Grpc-Metadata-Content-Type: application/grpc
Pragma: no-cache
Server: nginx
Strict-Transport-Security: max-age=63072000; includeSubdomains;
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
X-Xss-Protection: 1; mode=block

{
“service”: {
“service_id”: “/service_id/5b0a2717-cf67-42f0-ba48-5e580937e5f8”,
“service_name”: “proxysql-service”,
“node_id”: “/node_id/5283e9c5-8545-4245-a942-6ba7899ad6d4”,
“address”: “172.30.18.212”,
“port”: 6032
},
“proxysql_exporter”: {
“agent_id”: “/agent_id/33e91160-ca16-46f9-9da4-0963a138a547”,
“pmm_agent_id”: “/agent_id/04b9c01f-e53a-4786-aaf4-5bbb2607a5d9”,
“service_id”: “/service_id/5b0a2717-cf67-42f0-ba48-5e580937e5f8”,
“username”: “radmin”,
“push_metrics_enabled”: true,
“status”: “UNKNOWN”,
“log_level”: “warn”
}
}
DEBUG 2025-04-15 07:25:10.042982397Z: Result: &management.addProxySQLResult{Service:(*proxy_sql.AddProxySQLOKBodyService)(0xc0000a2e60)}
DEBUG 2025-04-15 07:25:10.042999707Z: Error:
ProxySQL Service added.
Service ID : /service_id/5b0a2717-cf67-42f0-ba48-5e580937e5f8
Service name: proxysql-service

The ProxySQL version is

ubuntu@ip-172-30-18-212:~$ proxysql --version
ProxySQL version 2.7.2-96-g5816a7d, codename Truls

The status of the added ProxySQL service in the PMM Inventory remains N/A and I can’t get any metrics for it

Will highly appreciate your help on fixing this

2

Hello @Harutyun_Grigoryan

Please check pmm-agent log for further investigation,

journalctl -u pmm-agent.service -f

3

Thanks @lalit.choudhary

All I see there is

ubuntu@ip-172-30-18-212:~$ journalctl -u pmm-agent.service -f
Apr 16 03:00:00 ip-172-30-18-212 pmm-agent[739]: time=“2025-04-16T03:00:00.325+00:00” level=error msg=“Failed to connect to 172.30.18.50:443: timeout.” component=client
Apr 16 03:00:17 ip-172-30-18-212 pmm-agent[739]: time=“2025-04-16T03:00:17.052+00:00” level=error msg=“Failed to connect to 172.30.18.50:443: timeout.” component=client
Apr 16 03:00:39 ip-172-30-18-212 pmm-agent[739]: time=“2025-04-16T03:00:39.900+00:00” level=error msg=“Failed to connect to 172.30.18.50:443: timeout.” component=client
Apr 16 03:01:03 ip-172-30-18-212 pmm-agent[739]: time=“2025-04-16T03:01:03.422+00:00” level=error msg=“Failed to connect to 172.30.18.50:443: timeout.” component=client
Apr 16 03:01:19 ip-172-30-18-212 pmm-agent[739]: time=“2025-04-16T03:01:19.619+00:00” level=error msg=“Failed to reserve port: no free port.” component=supervisor
Apr 16 03:01:19 ip-172-30-18-212 pmm-agent[739]: time=“2025-04-16T03:01:19.620+00:00” level=error msg=“Failed to reserve port: no free port.” component=supervisor
Apr 16 03:01:19 ip-172-30-18-212 pmm-agent[739]: time=“2025-04-16T03:01:19.620+00:00” level=error msg=“Failed to reserve port: no free port.” component=supervisor
Apr 16 03:01:20 ip-172-30-18-212 pmm-agent[739]: time=“2025-04-16T03:01:20.887+00:00” level=error msg=“Failed to reserve port: no free port.” component=supervisor
Apr 16 03:01:20 ip-172-30-18-212 pmm-agent[739]: time=“2025-04-16T03:01:20.887+00:00” level=error msg=“Failed to reserve port: no free port.” component=supervisor
Apr 16 03:01:20 ip-172-30-18-212 pmm-agent[739]: time=“2025-04-16T03:01:20.888+00:00” level=error msg=“Failed to reserve port: no free port.” component=supervisor

The connection errors at 3 AM are caused by Percona EC2 and ProxySQL being paused in the night and started at that time

4

Hello @Harutyun_Grigoryan

Apr 16 03:01:03 ip-172-30-18-212 pmm-agent[739]: time=“2025-04-16T03:01:03.422+00:00” level=error msg=“Failed to connect to 172.30.18.50:443: timeout.” component=client
Apr 16 03:01:19 ip-172-30-18-212 pmm-agent[739]: time=“2025-04-16T03:01:19.619+00:00” level=error msg=“Failed to reserve port: no free port.” component=supervisor

Seems like there is a connectivity issue to pmm-server, Please check connectivity to the pmm server using telnet for 443 port,

telnet 172.30.18.50 443

5

@lalit.choudhary

ubuntu@ip-172-30-18-212:~$ telnet 172.30.18.50 443
Trying 172.30.18.50...
Connected to 172.30.18.50.
Escape character is '^]'.
^]
telnet> 

quit
HTTP/1.1 400 Bad Request
Server: nginx
Date: Thu, 17 Apr 2025 13:48:10 GMT
Content-Type: text/html
Content-Length: 150
Connection: close
Strict-Transport-Security: max-age=63072000; includeSubdomains;

<html>
<head><title>400 Bad Request</title></head>
<body>
<center><h1>400 Bad Request</h1></center>
<hr><center>nginx</center>
</body>
</html>
Connection closed by foreign host.
ubuntu@ip-172-30-18-212:~$

The logs reflect that the ProxySQL and Percona EC2-s are restarted daily at 3 AM UTC
I get similar logs every day

6

Hello @Harutyun_Grigoryan

Sorry, I missed this. So, what’s the status now?

The logs reflect that the ProxySQL and Percona EC2-s are restarted daily at 3 AM UTC
I get similar logs every day

You should investigate with the admin to find the cause of the restart. But the expectation is that once proxysql is available and running, PMM should resume its monitoring.

I would suggest re-adding proxysql, and if you see any further issues, check the pmm-admin list output , pmm-inventory status and pmm-agent log for further investigation.