Multiple Errors from PMM in the MariaDB sql error log file

Percona Monitoring and Management (PMM)
1

Below are errors flooding our SQL error log file.

Source MariaDB: 10.4

2023-08-08 19:29:52 pmm_user[pmm_user] @ [192.168.1.1] ERROR 1146: Table ‘heartbeat.heartbeat’ doesn’t exist : SELECT UNIX_TIMESTAMP(ts), UNIX_TIMESTAMP(NOW(6)), server_id from heartbeat.heartbeat
2023-08-08 19:29:52 pmm_user[pmm_user] @ [192.168.1.1] ERROR 1286: Unknown storage engine ‘TOKUDB’ : SHOW ENGINE TOKUDB STATUS
2023-08-08 19:29:54 pmm_user[pmm_user] @ [192.168.1.1] ERROR 1193: Unknown system variable ‘slow_query_log_always_write_time’ : SELECT /* pmm-agent:slowlog / @@slow_query_log_always_write_time
2023-08-08 19:30:00 pmm_user[pmm_user] @ [192.168.1.1] ERROR 1109: Unknown table ‘QUERY_RESPONSE_TIME_READ’ in information_schema : SELECT TIME, COUNT, TOTAL FROM INFORMATION_SCHEMA.QUERY_RESPONSE_TIME_READ
2023-08-08 19:30:00 pmm_user[pmm_user] @ [192.168.1.1] ERROR 1109: Unknown table ‘QUERY_RESPONSE_TIME_WRITE’ in information_schema : SELECT TIME, COUNT, TOTAL FROM INFORMATION_SCHEMA.QUERY_RESPONSE_TIME_WRITE
2023-08-08 19:30:04 pmm_user[pmm_user] @ [192.168.1.1] ERROR 1193: Unknown system variable ‘slow_query_log_always_write_time’ : SELECT / pmm-agent:slowlog / @@slow_query_log_always_write_time
2023-08-08 19:30:10 pmm_user[pmm_user] @ [192.168.1.1] ERROR 1109: Unknown table ‘QUERY_RESPONSE_TIME_READ’ in information_schema : SELECT TIME, COUNT, TOTAL FROM INFORMATION_SCHEMA.QUERY_RESPONSE_TIME_READ
2023-08-08 19:30:10 pmm_user[pmm_user] @ [192.168.1.1] ERROR 1109: Unknown table ‘QUERY_RESPONSE_TIME_WRITE’ in information_schema : SELECT TIME, COUNT, TOTAL FROM INFORMATION_SCHEMA.QUERY_RESPONSE_TIME_WRITE
2023-08-08 19:30:14 pmm_user[pmm_user] @ [192.168.1.1] ERROR 1193: Unknown system variable ‘slow_query_log_always_write_time’ : SELECT / pmm-agent:slowlog / @@slow_query_log_always_write_time
2023-08-08 19:30:20 pmm_user[pmm_user] @ [192.168.1.1] ERROR 1109: Unknown table ‘QUERY_RESPONSE_TIME_READ’ in information_schema : SELECT TIME, COUNT, TOTAL FROM INFORMATION_SCHEMA.QUERY_RESPONSE_TIME_READ
2023-08-08 19:30:20 pmm_user[pmm_user] @ [192.168.1.1] ERROR 1109: Unknown table ‘QUERY_RESPONSE_TIME_WRITE’ in information_schema : SELECT TIME, COUNT, TOTAL FROM INFORMATION_SCHEMA.QUERY_RESPONSE_TIME_WRITE
2023-08-08 19:30:24 pmm_user[pmm_user] @ [192.168.1.1] ERROR 1193: Unknown system variable ‘slow_query_log_always_write_time’ : SELECT / pmm-agent:slowlog / @@slow_query_log_always_write_time
2023-08-08 19:30:30 pmm_user[pmm_user] @ [192.168.1.1] ERROR 1109: Unknown table ‘QUERY_RESPONSE_TIME_READ’ in information_schema : SELECT TIME, COUNT, TOTAL FROM INFORMATION_SCHEMA.QUERY_RESPONSE_TIME_READ
2023-08-08 19:30:30 pmm_user[pmm_user] @ [192.168.1.1] ERROR 1109: Unknown table ‘QUERY_RESPONSE_TIME_WRITE’ in information_schema : SELECT TIME, COUNT, TOTAL FROM INFORMATION_SCHEMA.QUERY_RESPONSE_TIME_WRITE
2023-08-08 19:30:34 pmm_user[pmm_user] @ [192.168.1.1] ERROR 1193: Unknown system variable ‘slow_query_log_always_write_time’ : SELECT / pmm-agent:slowlog / @@slow_query_log_always_write_time
2023-08-08 19:30:40 pmm_user[pmm_user] @ [192.168.1.1] ERROR 1109: Unknown table ‘QUERY_RESPONSE_TIME_READ’ in information_schema : SELECT TIME, COUNT, TOTAL FROM INFORMATION_SCHEMA.QUERY_RESPONSE_TIME_READ
2023-08-08 19:30:40 pmm_user[pmm_user] @ [192.168.1.1] ERROR 1109: Unknown table ‘QUERY_RESPONSE_TIME_WRITE’ in information_schema : SELECT TIME, COUNT, TOTAL FROM INFORMATION_SCHEMA.QUERY_RESPONSE_TIME_WRITE
2023-08-08 19:30:44 pmm_user[pmm_user] @ [192.168.1.1] ERROR 1193: Unknown system variable ‘slow_query_log_always_write_time’ : SELECT / pmm-agent:slowlog */ @@slow_query_log_always_write_time
2023-08-08 19:30:50 pmm_user[pmm_user] @ [192.168.1.1] ERROR 1109: Unknown table ‘QUERY_RESPONSE_TIME_READ’ in information_schema : SELECT TIME, COUNT, TOTAL FROM INFORMATION_SCHEMA.QUERY_RESPONSE_TIME_READ
2023-08-08 19:30:50 pmm_user[pmm_user] @ [192.168.1.1] ERROR 1109: Unknown table ‘QUERY_RESPONSE_TIME_WRITE’ in information_schema : SELECT TIME, COUNT, TOTAL FROM INFORMATION_SCHEMA.QUERY_RESPONSE_TIME_WRITE

2

It looks like the ‘SQL error log file’ is a plugin unique to MariaDB which logs SQL errors. This plugin, unfortunately, has no filtering capabilities.

The SQL errors you see are coming from PMM, attempting to query basic information about your system such as query response times, any tokudb engine stats, and replication heartbeat. There are also some typical system variables that MariaDB doesn’t appear to support.

I will pass this info along to our PMM team for a possible feature request to not run these types of queries against MariaDB. Until then, I’m afraid these queries will continue to execute.

3

Hi @matthewb

Thanks for the update.

They are consistently writing to the sql error log, thus it would be better if we could disable this option.

I can see the following error in the slow query log. Could you also look at these slow log related errors?

*2023-08-08 19:30:44 pmm_user[pmm_user] @ [192.168.1.1] ERROR 1193: Unknown system variable ‘slow_query_log_always_write_time’ : SELECT / pmm-agent:slowlog /

4

Unfortunately, disabling isn’t an option in PMM as of today. This is a feature that would need to be added to PMM, tested, QA’d, etc. I wouldn’t expect a fix within the next several weeks, possibly a month.

The error in your slow log is the same issue as the others, MariaDB doesn’t have that variable.

5

Got it @matthewb

I appreciate the thorough explanation. Could you kindly ask that we fix this problem internally, preferably for the next release 2.40 ?

6

I’ve created a JIRA ticket for the PMM team and referenced this forum post. It is out of my hands at this point as I’m not in a position to tell that team what to focus on. You are welcome to engage with us on a support contract which could absolutely help speed things along.

https://jira.percona.com/browse/PMM-12407

7

Thanks @matthewb for the ticket.

8

@matthewb Is there any way that we can turn off these checks at the PMM agent level?

9

Unfortunately, no. These queries come from various sub-systems within the PMM agent. Some are for the query analytics exporter, some are core to the mysqld exporter.

All of Percona’s software, including everything PMM related, is 100% opensource. You can pull down the exporter codebase and comment out all of these queries and recompile the exporter for your unique environment until this is fixed within PMM itself.

10

Many thanks for the information. @matthewb

Actually, I am not sure about the coding part :grinning:, but if you could please request it next one or two months if we can fix this issues would be great.