Pmm-admin logs too verbose into syslog

I would rather see only critical messages from ppm-agent in my system log.

Can I configure the level of verbosity in which it logs to syslog?

There’re way too many messages in /var/log/messages, like these:

Dec 11 17:56:39 pxc-3-euw-1 pmm-agent[822]: #033[36mINFO#033[0m[2020-12-11T17:56:39.387+00:00] time=“2020-12-11T17:56:39Z” level=info msg=“database version 8.0.20-11.3, distro Percona XtraDB Cluster (GPL), Release rel11, Revision d9fba7f, WSREP version 26.4.3” source=“slave_status.go:72” #033[36magentID#033[0m=/agent_id/86327eba-0c8c-4566-a5ac-abdf7db64c84 #033[36mcomponent#033[0m=agent-process #033[36mtype#033[0m=mysqld_exporter

Dec 11 17:56:39 pxc-3-euw-1 pmm-agent[822]: #033[36mINFO#033[0m[2020-12-11T17:56:39.393+00:00] time="2020-12-11T17:56:39Z" level=info msg="Successfully scraped status with query: SHOW SLAVE STATUS" source="slave_status.go:113" #033[36magentID#033[0m=/agent_id/86327eba-0c8c-4566-a5ac-abdf7db64c84 #033[36mcomponent#033[0m=agent-process #033[36mtype#033[0m=mysqld_exporter

Dec 11 17:56:49 pxc-3-euw-1 pmm-agent[822]: #033[36mINFO#033[0m[2020-12-11T17:56:49.387+00:00] time="2020-12-11T17:56:49Z" level=info msg="database version 8.0.20-11.3, distro Percona XtraDB Cluster (GPL), Release rel11, Revision d9fba7f, WSREP version 26.4.3" source="slave_status.go:72" #033[36magentID#033[0m=/agent_id/86327eba-0c8c-4566-a5ac-abdf7db64c84 #033[36mcomponent#033[0m=agent-process #033[36mtype#033[0m=mysqld_exporter

Dec 11 17:56:49 pxc-3-euw-1 pmm-agent[822]: #033[36mINFO#033[0m[2020-12-11T17:56:49.395+00:00] time="2020-12-11T17:56:49Z" level=info msg="Successfully scraped status with query: SHOW SLAVE STATUS" source="slave_status.go:113" #033[36magentID#033[0m=/agent_id/86327eba-0c8c-4566-a5ac-abdf7db64c84 #033[36mcomponent#033[0m=agent-process #033[36mtype#033[0m=mysqld_exporter

Dec 11 17:56:59 pxc-3-euw-1 pmm-agent[822]: #033[36mINFO#033[0m[2020-12-11T17:56:59.387+00:00] time="2020-12-11T17:56:59Z" level=info msg="database version 8.0.20-11.3, distro Percona XtraDB Cluster (GPL), Release rel11, Revision d9fba7f, WSREP version 26.4.3" source="slave_status.go:72" #033[36magentID#033[0m=/agent_id/86327eba-0c8c-4566-a5ac-abdf7db64c84 #033[36mcomponent#033[0m=agent-process #033[36mtype#033[0m=mysqld_exporter

Dec 11 17:56:59 pxc-3-euw-1 pmm-agent[822]: #033[36mINFO#033[0m[2020-12-11T17:56:59.391+00:00] time="2020-12-11T17:56:59Z" level=info msg="Successfully scraped status with query: SHOW SLAVE STATUS" source="slave_status.go:113" #033[36magentID#033[0m=/agent_id/86327eba-0c8c-4566-a5ac-abdf7db64c84 #033[36mcomponent#033[0m=agent-process #033[36mtype#033[0m=mysqld_exporter

Dec 11 17:57:00 pxc-3-euw-1 pmm-agent[822]: #033[36mINFO#033[0m[2020-12-11T17:57:00.001+00:00] Sending 50 buckets.                          #033[36magentID#033[0m=/agent_id/9098b93e-b65b-4e3f-a2d6-3c432ae32870 #033[36mcomponent#033[0m=agent-builtin #033[36mtype#033[0m=qan_mysql_slowlog_agent

Dec 11 17:57:09 pxc-3-euw-1 pmm-agent[822]: #033[36mINFO#033[0m[2020-12-11T17:57:09.386+00:00] time="2020-12-11T17:57:09Z" level=info msg="database version 8.0.20-11.3, distro Percona XtraDB Cluster (GPL), Release rel11, Revision d9fba7f, WSREP version 26.4.3" source="slave_status.go:72" #033[36magentID#033[0m=/agent_id/86327eba-0c8c-4566-a5ac-abdf7db64c84 #033[36mcomponent#033[0m=agent-process #033[36mtype#033[0m=mysqld_exporter

Dec 11 17:57:09 pxc-3-euw-1 pmm-agent[822]: #033[36mINFO#033[0m[2020-12-11T17:57:09.388+00:00] time="2020-12-11T17:57:09Z" level=info msg="Successfully scraped status with query: SHOW SLAVE STATUS" source="slave_status.go:113" #033[36magentID#033[0m=/agent_id/86327eba-0c8c-4566-a5ac-abdf7db64c84 #033[36mcomponent#033[0m=agent-process #033[36mtype#033[0m=mysqld_exporter

Dec 11 17:57:19 pxc-3-euw-1 pmm-agent[822]: #033[36mINFO#033[0m[2020-12-11T17:57:19.389+00:00] time="2020-12-11T17:57:19Z" level=info msg="database version 8.0.20-11.3, distro Percona XtraDB Cluster (GPL), Release rel11, Revision d9fba7f, WSREP version 26.4.3" source="slave_status.go:72" #033[36magentID#033[0m=/agent_id/86327eba-0c8c-4566-a5ac-abdf7db64c84 #033[36mcomponent#033[0m=agent-process #033[36mtype#033[0m=mysqld_exporter

Dec 11 17:57:19 pxc-3-euw-1 pmm-agent[822]: #033[36mINFO#033[0m[2020-12-11T17:57:19.392+00:00] time="2020-12-11T17:57:19Z" level=info msg="Successfully scraped status with query: SHOW SLAVE STATUS" source="slave_status.go:113" #033[36magentID#033[0m=/agent_id/86327eba-0c8c-4566-a5ac-abdf7db64c84 #033[36mcomponent#033[0m=agent-process #033[36mtype#033[0m=mysqld_exporter

Hi @split-brain,

pmm-agent sends all messages into stderr stream. Also it has no flags to exclude INFO messages from output.

1 Like

thanks @adivinho,

can I specify some other file for pmm-agent to log to?

1 Like

You may redirect output into any file.

/var/log/pmm-agent.log is specified in pmm-agent script.

1 Like

I’m on centos 8, so using systemd

couldn’t find the script with the option for loging to a file

after reading this I wonder if there is such option?

1 Like

same question

1 Like

The best way is to customize the log file

1 Like

Hi

You could set StandartError parameter in pmm-agent unit configuration file.

https://www.freedesktop.org/software/systemd/man/systemd.exec.html#StandardError=

2 Likes

Here is a screenshot of my tests

1 Like

thank you @adivinho, this solved my problem!

1 Like

Hi i tried to do the same thing but still logs are going to syslog.

[Unit]
Description=pmm-agent
After=time-sync.target network.target

[Service]
Type=simple
ExecStart=/usr/sbin/pmm-agent --config-file=/usr/local/percona/pmm2/config/pmm-agent.yaml
Restart=always
RestartSec=2s
StandardError=file:/var/log/pmm-agent.log
[Install]
WantedBy=multi-user.target

[root@dbsl01 log]# touch /var/log/pmm-agent.log
[root@dbsl01 log]# systemctl daemon-reload
[root@dbsl01 log]# systemctl restart pmm-agent
[root@dbsl01 log]# journalctl -f
-- Logs begin at Thu 2020-10-15 23:14:58 CDT. --
Jan 08 11:32:20<hostname> pmm-agent[28906]: INFO[2021-01-08T11:32:20.653-06:00] 2021-01-08T17:32:20.653Z        info        VictoriaMetrics/app/vmagent/main.go:111        started vmagent in 0.013 seconds  agentID=/agent_id/d89645e9-e902-4995-b4b4-2dfd176f7303 component=agent-process type=vm_agent
Jan 08 11:32:20 dbsl01.q2dc.local pmm-agent[28906]


-rw-r--r--  1 root  root             0 Jan  8 11:22 pmm-agent.log```
1 Like

Hi Dipesh,

Could you check documentation for your systemd version?
Maybe it’s required an empty line before the section [Install] or option StandardError has a different format.
But exactly your sample is working on my installation.

2 Likes

Hi Looks like this is not supported in centos 7 and also in systemd version 219. looks like systemd has to be 234 or above to make this work. is there are workaround that we can do.

the logs are too verbose for syslog

1 Like

Hi @dipeshacharya, could you create a task with explaining what’s expected and why in our jira and we will discuss about ability to configure the level of verbosity for pmm-agent or maybe change the level of most annoying message to debug.
Thank you.

1 Like

Hi @nurlan

here is the jira for it

  1. PMM-7326
1 Like

Thank you, @dipeshacharya

1 Like

We have this same problem. It is filling up /var/log on every system we have after upgrading to

# pmm-agent --version
ProjectName: pmm-agent
Version: 2.14.0
PMMVersion: 2.14.0
Timestamp: 2021-01-28 12:36:49 (UTC)
FullCommit: b9c1374a8b796bb6de73db65081fd8fd52c56eef

We need this fixed NOW.

This is from one server over the past two days:

# grep pmm-agent messages | awk '{print $1 $2}' | uniq -c
 134138 Feb21
 116785 Feb22
1 Like

The solution to this was to modify one of the queries in /usr/local/percona/pmm2/collectors/custom-queries/mysql/high-resolution/queries-mysqld-group-replication.yml to add a DISTINCT clause. The large majority of the errors being logged were HUGE error messages to be written to /var/log/messages. By adding the DISTINCT clause, we eliminated this class of errors without breaking the functionality of the telemetry itself.

Here is the change:

mysql_perf_schema_replication_group_worker_5:
  query: "/*!50700 SELECT DISTINCT conn_status.channel_name as channel_name,

This only has an effect, of course, if you are using more than one slave_parallel_worker, which we are.

2 Likes