Description:
We have an issue with scraping mysql_global_variables_read_only using pmm-client container. We have pmm-client Slow query log source configured but for some reason scraping has an issue after we enabled performance_schema on the database. Yesterday, we increased performance_schema_digests_size (not related to this issue), and after restarting the database scraping worked for 6 hours and then completely stopped. Metrics resolution on the server side is set to rare.
Version:
pmm-client: Version: 2.41.1
Database: Ver 8.0.29-21 for Linux on x86_64 (Percona Server (GPL), Release 21, Revision c59f87d2854)
Does anyone have any idea how to debug the issue?
Rare equates to, 300s, 180s, 60s for low, medium, and high resolution. Have you inspected the PMM agent logs on the host to see if there are any errors?
When I set up logging to trace by setting PMM_AGENT_TRACE=true then we have thousands of logs in a few seconds. Do you have some keywords to look for?
Keywords would be ‘error’, or ‘warning’, or ‘fail’.
Unfortunately, logs are completely non-readable due to the number of rows it generates. When I grep error, warning, or fail it gives me also everything from the msg field where are all statements parsed from the slow log. Do you have some idea how to improve grep? Maybe include another field instead msg?
grep ERROR /path/to/log should only show you lines containing the word ERROR.
I checked logs but it includes ERROR only in case of SQL query. Nothing pmm-client scrape related. When I turn off performance_schema on the database, the scrape works as expected.
The SQL queries are pmm scraping data. What SQL is producing ERRORs? Please verify pmm-admin status that you are using the slow query log exporter.
It is not ERRORs itself, it’s queries that include ERROR in statement. pmm-admin status shows mysql_slowlog_agent Running 0. Can you tell me how pmm-client getting mysql_global_variables_read_only? By which query?
This is ‘SHOW GLOBAL VARIABLES’, and the mysqld_exporter parses each of the 500 values.
We used ngrep to catch communication and we can see that pmm-clinet receives information so there must be an issue between the client and server or the client itself. Any ideas on how to debug this issue further?