No data from client

[sarasx]

I’m trying to setup MySQL monitoring, but it doesn’t seem to work.

After setting up pmm agent on the db node, the node appears on the pmm dashboard but there is no data, the only thing I’m getting is a System Information message

No pmm-agent running on this node

which after some time changes to correct system data (hostname, cpu, ram, etc…).
On the db node:
pmm-admin status

Agent ID: /agent_id/e0a6d26f-08e8-4213-abc6-f219062df39b
Node ID : /node_id/b8fd9824-5e65-4e6f-8b8e-0eceb1233c53

PMM Server:
        URL    : https://localhost:11443/ (NOTE: this is an ssh tunnel)
        Version: 2.11.1

PMM Client:
        Connected        : true
        Time drift       : 1.280067ms
        Latency          : 1.978329ms
        pmm-admin version: 2.11.1
        pmm-agent version: 2.11.1
Agents:
        /agent_id/4b05f2a7-bbac-4cff-b1a6-6a9bc6cbee05 mysql_perfschema_agent Running
        /agent_id/91da16f9-c8d0-41ee-97c3-d62df282c5cc node_exporter Running
        /agent_id/b57a494a-fee7-4c3d-95f4-f498468d67fa mysqld_exporter Running

pmm-admin list

Service type  Service name         Address and port  Service ID
MySQL         myServiceName         127.0.0.1:3306    /service_id/8c508bf3-bdd8-4971-b519-1b5cb0e7f113

Agent type                  Status     Agent ID                                        Service ID
pmm_agent                   Connected  /agent_id/e0a6d26f-08e8-4213-abc6-f219062df39b
node_exporter               Running    /agent_id/91da16f9-c8d0-41ee-97c3-d62df282c5cc
mysqld_exporter             Running    /agent_id/b57a494a-fee7-4c3d-95f4-f498468d67fa  /service_id/8c508bf3-bdd8-4971-b519-1b5cb0e7f113
mysql_perfschema_agent      Running    /agent_id/4b05f2a7-bbac-4cff-b1a6-6a9bc6cbee05  /service_id/8c508bf3-bdd8-4971-b519-1b5cb0e7f113

pmm can be seen performing queries on the database every few seconds

what logs should i attach?

[Vadim_Yalovets]

Hi Sarasx,

Could you perform next command on the monitor node?

curl -k -s -u pmm:/agent_id/b57a494a-fee7-4c3d-95f4-f498468d67fa http://127.0.0.1:42001/metrics

Also please check if it’s possible to get metrics on pmm-server directly by the same command. But please replace 127.0.0.1 by node’s IP.

[sarasx]

Output on db node

curl -k -u pmm:/agent_id/b57a494a-fee7-4c3d-95f4-f498468d67fa http://127.0.0.1:42001/metrics
curl: (7) Failed to connect to 127.0.0.1 port 42001: Connection refused

I removed -s to see error output, same on pmm-server

[Vadim_Yalovets]

Just please make sure that it’s correct port that is used by mysqld_exporter

If port is really 42001 then access to this port is rejected on your node.

[sarasx]

Here’s the output

root     10741  0.0  0.1 716036 46560 ?        Sl   Oct21   0:06 /usr/local/percona/pmm2/exporters/mysqld_exporter --collect.auto_increment.columns --collect.binlog_size --collect.custom_query.hr --collect.custom_query.hr.directory=/usr/local/percona/pmm2/collectors/custom-queries/mysql/high-resolution --collect.custom_query.lr --collect.custom_query.lr.directory=/usr/local/percona/pmm2/collectors/custom-queries/mysql/low-resolution --collect.custom_query.mr --collect.custom_query.mr.directory=/usr/local/percona/pmm2/collectors/custom-queries/mysql/medium-resolution --collect.engine_innodb_status --collect.engine_tokudb_status --collect.global_status --collect.global_variables --collect.heartbeat --collect.info_schema.clientstats --collect.info_schema.innodb_cmp --collect.info_schema.innodb_cmpmem --collect.info_schema.innodb_metrics --collect.info_schema.innodb_tablespaces --collect.info_schema.processlist --collect.info_schema.query_response_time --collect.info_schema.tables --collect.info_schema.tablestats --collect.info_schema.userstats --collect.perf_schema.eventsstatements --collect.perf_schema.eventswaits --collect.perf_schema.file_events --collect.perf_schema.file_instances --collect.perf_schema.indexiowaits --collect.perf_schema.tableiowaits --collect.perf_schema.tablelocks --collect.slave_status --collect.standard.go --collect.standard.process --exporter.conn-max-lifetime=55s --exporter.global-conn-pool --exporter.max-idle-conns=3 --exporter.max-open-conns=3 --web.listen-address=:42002

particullary

--web.listen-address=:42002

Running

curl -k -u pmm:/agent_id/b57a494a-fee7-4c3d-95f4-f498468d67fa http://127.0.0.1:42002/metrics

on the db node gives output (mysql_perf_schema_file_instances_bytes{event_name="wait/io/file/innodb/innodb_data_file",file_name="<redacted>",mode="read"} 4.01408e+06 and so on), same on pmm-server when runnin with node IP

[Vadim_Yalovets]

So pmm-server is able to fetch metrics from the db node.

But no data are displayed on mysql dashboards. Am I right?

Could you check targets states for this mysqld_exporter on Prometheus UI?

https://your-pmm-server/prometheus/targets

e.g.

[sarasx]

So pmm-server is able to fetch metrics from the db node.

But no data are displayed on mysql dashboards. Am I right?

Yes

It seems pmm-server thinks the DB node is on localhost.

The node was registered running the following command on the node:

sudo pmm-admin config --server-insecure-tls --server-url=https://admin:<password>@localhost:11443

The db node can’t talk directly with the pmm-server and has to go throught an ssh tunnel, but the pmm-server can talk to the node.

[Vadim_Yalovets]

The script couldn’t correctly detect your node IP.

Please specify it in configuration command.

e.g.

sudo pmm-admin config --server-insecure-tls --server-url=https://admin:<password>@localhost:11443 172.17.0.1 generic mynode1

[sarasx]

After reruning pmm-admin config with an IP parameter and rerunning pmm-admin add mysql I’m getting data

[steve.hoffman]

just for anyone that stumbles on this across the internet you can run pmm-admin config --help or pmm-admin add <mysql|postgresql|mongodb> --help and see what we autodetected as the IP of the system, if you don’t specify that positional argument, we’ll default to what we detected. It’s on our roadmap to enhance that or at least make it more obvious what we will use.