Not the answer you need?
Register and ask your own question!

No data from client

sarasxsarasx Current User Role Contributor

I'm trying to setup MySQL monitoring, but it doesn't seem to work.

After setting up pmm agent on the db node, the node appears on the pmm dashboard but there is no data, the only thing I'm getting is a System Information message

No pmm-agent running on this node

which after some time changes to correct system data (hostname, cpu, ram, etc...).

On the db node:

pmm-admin status

Agent ID: /agent_id/e0a6d26f-08e8-4213-abc6-f219062df39b
Node ID : /node_id/b8fd9824-5e65-4e6f-8b8e-0eceb1233c53

PMM Server:
        URL    : https://localhost:11443/ (NOTE: this is an ssh tunnel)
        Version: 2.11.1

PMM Client:
        Connected        : true
        Time drift       : 1.280067ms
        Latency          : 1.978329ms
        pmm-admin version: 2.11.1
        pmm-agent version: 2.11.1
Agents:
        /agent_id/4b05f2a7-bbac-4cff-b1a6-6a9bc6cbee05 mysql_perfschema_agent Running
        /agent_id/91da16f9-c8d0-41ee-97c3-d62df282c5cc node_exporter Running
        /agent_id/b57a494a-fee7-4c3d-95f4-f498468d67fa mysqld_exporter Running

pmm-admin list

Service type  Service name         Address and port  Service ID
MySQL         myServiceName         127.0.0.1:3306    /service_id/8c508bf3-bdd8-4971-b519-1b5cb0e7f113

Agent type                  Status     Agent ID                                        Service ID
pmm_agent                   Connected  /agent_id/e0a6d26f-08e8-4213-abc6-f219062df39b
node_exporter               Running    /agent_id/91da16f9-c8d0-41ee-97c3-d62df282c5cc
mysqld_exporter             Running    /agent_id/b57a494a-fee7-4c3d-95f4-f498468d67fa  /service_id/8c508bf3-bdd8-4971-b519-1b5cb0e7f113
mysql_perfschema_agent      Running    /agent_id/4b05f2a7-bbac-4cff-b1a6-6a9bc6cbee05  /service_id/8c508bf3-bdd8-4971-b519-1b5cb0e7f113

pmm can be seen performing queries on the database every few seconds

what logs should i attach?

Best Answer

Answers

  • adivinhoadivinho Percona Percona Staff Role

    Hi Sarasx,

    Could you perform next command on the monitor node?

    curl -k -s -u pmm:/agent_id/b57a494a-fee7-4c3d-95f4-f498468d67fa http://127.0.0.1:42001/metrics
    

    Also please check if it's possible to get metrics on pmm-server directly by the same command. But please replace 127.0.0.1 by node's IP.

  • sarasxsarasx Current User Role Contributor
    edited October 22

    Output on db node

    curl -k -u pmm:/agent_id/b57a494a-fee7-4c3d-95f4-f498468d67fa http://127.0.0.1:42001/metrics
    curl: (7) Failed to connect to 127.0.0.1 port 42001: Connection refused
    

    I removed -s to see error output, same on pmm-server

  • adivinhoadivinho Percona Percona Staff Role

    Just please make sure that it's correct port that is used by mysqld_exporter

    If port is really 42001 then access to this port is rejected on your node.


  • sarasxsarasx Current User Role Contributor

    Here's the output

    root     10741  0.0  0.1 716036 46560 ?        Sl   Oct21   0:06 /usr/local/percona/pmm2/exporters/mysqld_exporter --collect.auto_increment.columns --collect.binlog_size --collect.custom_query.hr --collect.custom_query.hr.directory=/usr/local/percona/pmm2/collectors/custom-queries/mysql/high-resolution --collect.custom_query.lr --collect.custom_query.lr.directory=/usr/local/percona/pmm2/collectors/custom-queries/mysql/low-resolution --collect.custom_query.mr --collect.custom_query.mr.directory=/usr/local/percona/pmm2/collectors/custom-queries/mysql/medium-resolution --collect.engine_innodb_status --collect.engine_tokudb_status --collect.global_status --collect.global_variables --collect.heartbeat --collect.info_schema.clientstats --collect.info_schema.innodb_cmp --collect.info_schema.innodb_cmpmem --collect.info_schema.innodb_metrics --collect.info_schema.innodb_tablespaces --collect.info_schema.processlist --collect.info_schema.query_response_time --collect.info_schema.tables --collect.info_schema.tablestats --collect.info_schema.userstats --collect.perf_schema.eventsstatements --collect.perf_schema.eventswaits --collect.perf_schema.file_events --collect.perf_schema.file_instances --collect.perf_schema.indexiowaits --collect.perf_schema.tableiowaits --collect.perf_schema.tablelocks --collect.slave_status --collect.standard.go --collect.standard.process --exporter.conn-max-lifetime=55s --exporter.global-conn-pool --exporter.max-idle-conns=3 --exporter.max-open-conns=3 --web.listen-address=:42002
    

    particullary

    --web.listen-address=:42002
    

    Running

    curl -k -u pmm:/agent_id/b57a494a-fee7-4c3d-95f4-f498468d67fa http://127.0.0.1:42002/metrics
    

    on the db node gives output (mysql_perf_schema_file_instances_bytes{event_name="wait/io/file/innodb/innodb_data_file",file_name="<redacted>",mode="read"} 4.01408e+06 and so on), same on pmm-server when runnin with node IP

  • adivinhoadivinho Percona Percona Staff Role

    So pmm-server is able to fetch metrics from the db node.

    But no data are displayed on mysql dashboards. Am I right?

    Could you check targets states for this mysqld_exporter on Prometheus UI?

    https://your-pmm-server/prometheus/targets

    e.g.


  • sarasxsarasx Current User Role Contributor

    So pmm-server is able to fetch metrics from the db node.

    But no data are displayed on mysql dashboards. Am I right?

    Yes

    It seems pmm-server thinks the DB node is on localhost.

    The node was registered running the following command on the node:

    sudo pmm-admin config --server-insecure-tls --server-url=https://admin:<password>@localhost:11443
    

    The db node can't talk directly with the pmm-server and has to go throught an ssh tunnel, but the pmm-server can talk to the node.


  • sarasxsarasx Current User Role Contributor

    After reruning pmm-admin config with an IP parameter and rerunning pmm-admin add mysql I'm getting data

  • steve.hoffmansteve.hoffman Percona Percona Staff Role

    just for anyone that stumbles on this across the internet you can run pmm-admin config --help or pmm-admin add <mysql|postgresql|mongodb> --help and see what we autodetected as the IP of the system, if you don't specify that positional argument, we'll default to what we detected. It's on our roadmap to enhance that or at least make it more obvious what we will use.

Sign In or Register to comment.

MySQL, InnoDB, MariaDB and MongoDB are trademarks of their respective owners.
Copyright ©2005 - 2020 Percona LLC. All rights reserved.