# Scrape MySQL with performance\_schema ON **URL:** \ **Category:** Percona Monitoring and Management (PMM)\ **Tags:** pmm, mysql\ **Created:** [March 20, 2024, 11:46am UTC](https://forums.percona.com/t/scrape-mysql-with-performance-schema-on/29251 "2024-03-20T11:46:27Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![Jakub\_Noga](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/jakub_noga/32/12200_2.png) [@Jakub\_Noga](https://forums.percona.com/u/Jakub_Noga)\ **Post date:** [March 20, 2024, 11:46am UTC](https://forums.percona.com/t/scrape-mysql-with-performance-schema-on/29251/1 "2024-03-20T11:46:28Z")
## Description: We have an issue with scraping `mysql_global_variables_read_only` using pmm-client container. We have pmm-client **Slow query log** source configured but for some reason scraping has an issue after we enabled performance\_schema on the database. Yesterday, we increased performance\_schema\_digests\_size (not related to this issue), and after restarting the database scraping worked for 6 hours and then completely stopped. Metrics resolution on the server side is set to `rare`. ## Version: pmm-client: Version: 2.41.1 Database: Ver 8.0.29-21 for Linux on x86\_64 (Percona Server (GPL), Release 21, Revision c59f87d2854) Does anyone have any idea how to debug the issue? ---
**Author:** ![matthewb](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/matthewb/32/34_2.png) [@matthewb](https://forums.percona.com/u/matthewb)\ **Post date:** [March 20, 2024, 2:05pm UTC](https://forums.percona.com/t/scrape-mysql-with-performance-schema-on/29251/2 "2024-03-20T14:05:12Z")
Rare equates to, 300s, 180s, 60s for low, medium, and high resolution. Have you inspected the PMM agent logs on the host to see if there are any errors? ---
**Author:** ![Jakub\_Noga](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/jakub_noga/32/12200_2.png) [@Jakub\_Noga](https://forums.percona.com/u/Jakub_Noga)\ **Post date:** [March 21, 2024, 8:40am UTC](https://forums.percona.com/t/scrape-mysql-with-performance-schema-on/29251/3 "2024-03-21T08:40:58Z")
When I set up logging to trace by setting `PMM_AGENT_TRACE=true` then we have thousands of logs in a few seconds. Do you have some keywords to look for? ---
**Author:** ![matthewb](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/matthewb/32/34_2.png) [@matthewb](https://forums.percona.com/u/matthewb)\ **Post date:** [March 21, 2024, 1:27pm UTC](https://forums.percona.com/t/scrape-mysql-with-performance-schema-on/29251/4 "2024-03-21T13:27:39Z")
Keywords would be ‘error’, or ‘warning’, or ‘fail’. ---
**Author:** ![Jakub\_Noga](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/jakub_noga/32/12200_2.png) [@Jakub\_Noga](https://forums.percona.com/u/Jakub_Noga)\ **Post date:** [March 25, 2024, 2:14pm UTC](https://forums.percona.com/t/scrape-mysql-with-performance-schema-on/29251/5 "2024-03-25T14:14:58Z")
Unfortunately, logs are completely non-readable due to the number of rows it generates. When I grep `error`, `warning`, or `fail` it gives me also everything from the `msg` field where are all statements parsed from the slow log. Do you have some idea how to improve grep? Maybe include another field instead msg? ---
**Author:** ![matthewb](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/matthewb/32/34_2.png) [@matthewb](https://forums.percona.com/u/matthewb)\ **Post date:** [March 25, 2024, 3:32pm UTC](https://forums.percona.com/t/scrape-mysql-with-performance-schema-on/29251/6 "2024-03-25T15:32:09Z")
`grep ERROR /path/to/log` should only show you lines containing the word ERROR. ---
**Author:** ![Jakub\_Noga](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/jakub_noga/32/12200_2.png) [@Jakub\_Noga](https://forums.percona.com/u/Jakub_Noga)\ **Post date:** [March 28, 2024, 10:20am UTC](https://forums.percona.com/t/scrape-mysql-with-performance-schema-on/29251/7 "2024-03-28T10:20:00Z")
I checked logs but it includes ERROR only in case of SQL query. Nothing pmm-client scrape related. When I turn off performance\_schema on the database, the scrape works as expected. ---
**Author:** ![matthewb](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/matthewb/32/34_2.png) [@matthewb](https://forums.percona.com/u/matthewb)\ **Post date:** [March 28, 2024, 12:33pm UTC](https://forums.percona.com/t/scrape-mysql-with-performance-schema-on/29251/8 "2024-03-28T12:33:41Z")
The SQL queries are pmm scraping data. What SQL is producing ERRORs? Please verify `pmm-admin status` that you are using the slow query log exporter. ---
**Author:** ![Jakub\_Noga](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/jakub_noga/32/12200_2.png) [@Jakub\_Noga](https://forums.percona.com/u/Jakub_Noga)\ **Post date:** [April 10, 2024, 7:50am UTC](https://forums.percona.com/t/scrape-mysql-with-performance-schema-on/29251/9 "2024-04-10T07:50:43Z")
It is not ERRORs itself, it’s queries that include ERROR in statement. `pmm-admin status` shows `mysql_slowlog_agent Running 0`. Can you tell me how pmm-client getting `mysql_global_variables_read_only`? By which query? ---
**Author:** ![matthewb](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/matthewb/32/34_2.png) [@matthewb](https://forums.percona.com/u/matthewb)\ **Post date:** [April 17, 2024, 4:10pm UTC](https://forums.percona.com/t/scrape-mysql-with-performance-schema-on/29251/10 "2024-04-17T16:10:45Z")
> [@Jakub\_Noga](#): > > Can you tell me how pmm-client getting `mysql_global_variables_read_only`? By which query? This is ‘SHOW GLOBAL VARIABLES’, and the mysqld\_exporter parses each of the 500 values. ---
**Author:** ![Jakub\_Noga](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/jakub_noga/32/12200_2.png) [@Jakub\_Noga](https://forums.percona.com/u/Jakub_Noga)\ **Post date:** [April 25, 2024, 6:07am UTC](https://forums.percona.com/t/scrape-mysql-with-performance-schema-on/29251/11 "2024-04-25T06:07:51Z")
We used ngrep to catch communication and we can see that pmm-clinet receives information so there must be an issue between the client and server or the client itself. Any ideas on how to debug this issue further? ---
**Author:** ![Jakub\_Noga](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/jakub_noga/32/12200_2.png) [@Jakub\_Noga](https://forums.percona.com/u/Jakub_Noga)\ **Post date:** [September 23, 2024, 9:09am UTC](https://forums.percona.com/t/scrape-mysql-with-performance-schema-on/29251/12 "2024-09-23T09:09:58Z")
Hello, just FYI, the issue was caused by OOM killer. The increase of MEM on the container fixed the issue. Best regards, Jakub Noga.