PMM2 mysql exporter change port after each delete/add

mlaurent · December 30, 2020, 4:26pm

Hello,

When I add the mysql export on my client, the port changes each time : 42001 then 42002 then 42003 …

Prometheus manages to make the connection to get the metrics, only with port 42002, on the other ports I get the error “context deadline expected”.

On my firewall, ports 42000 to 50000 are open.

Example of command :

pmm-admin add mysql --server-url=_COPY15@myhttpsserver.com --query-source=slowlog --username=example --password=example

==> if I run a ps aux | grep pmm, the service runs on port 42001

pmm-admin remove mysql serverName-mysql

pmm-admin add mysql --server-url=_COPY15@myhttpsserver.com --query-source=slowlog --username=example --password=example

==> if I run a ps aux | grep pmm, the service runs on port 42002

Someone manages to reproduce it? I don’t know if it’s a bug, or a configuration problem on my side.

Have a nice day

adivinho · December 30, 2020, 4:41pm

Hi Laurent,

It’s possible to monitor few services by the same pmm-agent. Each new exporter will be start on an available 420xx port.

Could you check if any errors are shown in pmm agent log?

/var/log/pmm-agent.log

mlaurent · December 30, 2020, 4:37pm

Hi,

I have this error in the logs for November 2nd :

Nov 2 18:00:17 dba pmm-agent[2922]: #033[36mINFO#033[0m[2020-11-02T18:00:17.051+01:00] time="2020-11-02T18:00:17+01:00" level=info msg="Successfully scraped status with query: SHOW ALL SLAVES STATUS" source="slave_status.go:113" #033[36magentID#033[0m=/agent_id/be83475c-e4b3-4e80-8396-cdc77ecc224c #033[36mcomponent#033[0m=agent-process #033[36mtype#033[0m=mysqld_exporter
Nov 2 18:00:17 dba pmm-agent[2922]: #033[36mINFO#033[0m[2020-11-02T18:00:17.051+01:00] time="2020-11-02T18:00:17+01:00" level=error msg="Error scraping for collect.custom_query.hr: mysql_perf_schema_replication_group_worker_5:error running query on database: mysql_perf_schema_replication_group_worker_5, Error 1146: Table 'performance_schema.replication_connection_status' doesn't exist:mysql_perf_schema_replication_group:error running query on database: mysql_perf_schema_replication_group, Error 1193: Unknown system variable 'server_uuid':mysql_perf_schema_replication_group_5:error running query on database: mysql_perf_schema_replication_group_5, Error 1193: Unknown system variable 'server_uuid':mysql_perf_schema:error running query on database: mysql_perf_schema, Error 1193: Unknown system variable 'server_uuid':mysql_perf_schema_5:error running query on database: mysql_perf_schema_5, Error 1193: Unknown system variable 'server_uuid':mysql_perf_schema_replication_group_worker:error running query on database: mysql_perf_schema_replication_group_worker, Error 1305: FUNCTION GTID_SUBTRACT does not exist" source="exporter.go:116" #033[36magentID#033[0m=/agent_id/be83475c-e4b3-4e80-8396-cdc77ecc224c #033[36mcomponent#033[0m=agent-process #033[36mtype#033[0m=mysqld_exporter
The agent rebooted, it was running on port 42002 and I was receiving logs. It rebooted at 6pm last night on port 42001, and I'm not receiving anything anymore.
Here are the current logs, no apparent error:
Nov 3 16:19:00 dba pmm-agent[2922]: #033[36mINFO#033[0m[2020-11-03T16:19:00.001+01:00] Sending 66 buckets.              #033[36magentID#033[0m=/agent_id/58660e56-3e9a-4522-92e6-b07b2395b828 #033[36mcomponent#033[0m=agent-builtin #033[36mtype#033[0m=qan_mysql_slowlog_agent
Nov 3 16:20:00 dba pmm-agent[2922]: #033[36mINFO#033[0m[2020-11-03T16:20:00.001+01:00] Sending 92 buckets.              #033[36magentID#033[0m=/agent_id/58660e56-3e9a-4522-92e6-b07b2395b828 #033[36mcomponent#033[0m=agent-builtin #033[36mtype#033[0m=qan_mysql_slowlog_agent
Nov 3 16:21:00 dba pmm-agent[2922]: #033[36mINFO#033[0m[2020-11-03T16:21:00.000+01:00] Sending 30 buckets.              #033[36magentID#033[0m=/agent_id/58660e56-3e9a-4522-92e6-b07b2395b828 #033[36mcomponent#033[0m=agent-builtin #033[36mtype#033[0m=qan_mysql_slowlog_agent

I have my firewall open on the following ports : 42000:42020/tcp ALLOW Anywhere

Thanks