QAN showing query which are not running during the time range selected

Percona Monitoring and Management (PMM) PMM 3.x
, ,
1

Hi All,

I’m currently facing a rather confusing situation.

Issue:

While reviewing the Query Analytics (QAN) in PMM3 (and also in PMM 2.x), we noticed that when selecting a specific time range, some queries are being displayed that we are confident did not run during that selected timeframe.

For example -

image
image1920×709 113 KB

In the screenshot below, we selected a 5-minute window to view the queries executed during that time. However, the marked DELETE queries appear in the list — even though we are certain these queries only run during our scheduled maintenance window around 6:30 PM.

Environment details -
PMM version : 3.1.0 (Same issue with older 2.4xx as well. That’s why we upgraded)
Postgres version -PostgreSQL 16.6
pg_stat_statement : 1.10

We use pg_stat_statement and database servers has PMM client installed. PMM client version - 3.1.0.

Also, one more question - What is load here? as i do not find clear info about it. Does it mean that the query was executed and caused that load?

We would appreciate any insights or guidance on why these queries are showing up outside of their actual execution window.

Thank you.

2

Hello @aniruddha.deshpande,
I’ll let a PMM dev chime in about the query window issue, and I’ll explain that “Load” is simply “Query Count” x “Query Time”. Take your first query as example: 9.6 queries per second * 48.0 sec avg exec time = 460.8 load.

3

Hi @aniruddha.deshpande,

Could you please check if this is coming from pg_stat_statements itself or, for some reason, from PMM?

I think the simplest way to check is:
If you can see the mentioned queries appearing at a 5-minute interval (and you’re sure they aren’t triggered)

  1. Log in to a PG client
  2. Run following query:
    SELECT * FROM pg_stat_statements;

Let me know if you’re able to find such a query in the output.

Thank you.

4

Hello @matthewb ,

Thank you for your quick reply and clearing the air around LOAD.

5

Hello @Jiri_Ctvrtka1 ,

Thank you for your response.

If I understand you correctly, then yes — we have already verified this through pg_stat_statements by sampling at 1-minute intervals. During the specified time window, these queries were not executed. We also monitored over an extended period and confirmed that these queries were not executed during that time either.

6

Hi @aniruddha.deshpande,
Can you check how many records do you have in pg_stat_statements view? Is it bigger then 5000?

7

Hello @nurlan ,

yes.. we do have more than 5000.

PROD: postgres=# select count(*) from pg_stat_statements;
count

9490
(1 row)

PROD: postgres=# SELECT stats_reset
FROM pg_stat_statements_info;
stats_reset

2025-04-22 14:53:34.394045+02
(1 row)

some more info-

  1. As we saw this issue, we tried resetting the pg_stat_statements. Thats why you see the the reset data as recent as 22.04.
  2. Also I tried setting retention on PMM to 1 day to flush out old data.

This is production server so it does have lot of queries running on it.

8

Thank you,
we have reported bug about this behavior which is planned for fix in PMM 3.2.0

1 Like
9

Hello @nurlan ,

Thanks for the update! Glad to hear it’s been reported — looking forward to the fix in PMM 3.2.0.
Appreciate the support!

10

Hi @nurlan ,

Hope you are well.
May I kindly ask if there is an approximate timeline for the release of PMM 3.2.0? It would help us plan accordingly on our end.

11

It’s approximately two weeks from now, but no commitment :slight_smile: