Native Solution for Per-Query Latency Percentiles (P90/P95/P99) at DB Level in MySQL 8.0/8.4 with PMM — Request for Guidance

Percona Monitoring and Management (PMM) PMM 2.x
1

Dear Percona Support / Engineering Team,

We are currently investigating a significant observability gap and would like your guidance on whether a native or PMM-integrated solution exists.

pmm-server version: 2.39.0
mysql version: Mysql 8 / 8.4

We were checking on the “MySQL Query Response Time Details” dashboard in PMM, which was powered by the query_response_time plugin (available in Percona Server 5.6/5.7). This dashboard provided response time distribution histograms.

However, as confirmed in your official documentation (PMM MySQL Query Response Time Details), this dashboard and the underlying plugin have been removed in MySQL/Percona Server 8.0+.

As a result, DB-side mechanism to observe P90/P95/P99 query latencies. We are entirely dependent on application teams to provide query latency data, which:

  • Creates a bottleneck during incident response and RCA

  • Prevents us from proactively identifying slow query regressions

  • Makes it impossible to set DB-side SLOs on query latency

What We Have Tried So Far

1. PMM QAN (Performance Schema source)
QAN provides average query time and max query time per digest, but does not natively expose P90/P95/P99 percentile columns in its UI or standard dashboards.

2. performance_schema.events_statements_histogram_global

SELECT
ROUND(BUCKET_QUANTILE * 100, 1)AS percentile,
ROUND(BUCKET_TIMER_HIGH / 1000000000, 3)AS latency_ms
FROM performance_schema.events_statements_histogram_global
WHERE BUCKET_QUANTILE >= 0.95
ORDER BYBUCKET_QUANTILELIMIT 1;

This works but is global only , it aggregates all queries together and does not give per-query-digest percentiles. It is also cumulative, making time-windowed trending difficult.

3. QUANTILE_95 / QUANTILE_99 columns in events_statements_summary_by_digest

SELECT
SCHEMA_NAME,
LEFT(DIGEST_TEXT, 80) AS query,
COUNT_STARAS calls,
ROUND(QUANTILE_95 / 1000000000, 3)AS p95_ms,
ROUND(QUANTILE_99 / 1000000000, 3)AS p99_ms
FROM performance_schema.events_statements_summary_by_digest
WHERE SCHEMA_NAME IS NOT NULL AND COUNT_STAR > 100
ORDER BYQUANTILE_99DESC LIMIT 20;

This gives per-digest percentiles natively in MySQL 8.0+, which is promising. However, it is still cumulative and we are unsure of the best way to integrate this into PMM dashboards as a time-series metric.

4. sys.statements_with_runtimes_in_95th_percentile
Useful for ad-hoc investigation but not suitable for continuous monitoring or alerting.

We would appreciate your guidance on the following:

  1. Is there a native PMM 2/3.x dashboard or built-in panel that already visualizes P90/P95/P99 per query digest using QUANTILE_95/QUANTILE_99 from events_statements_summary_by_digest? If not, is this on the PMM roadmap?

  2. Does PMM’s mysqld_exporter already scrape QUANTILE_95/QUANTILE_99 columns from events_statements_summary_by_digest and expose them as Prometheus metrics? If yes, what are the metric names so we can build Grafana panels on top of them?

  3. What is the recommended approach for time-windowed P95/P99 metrics given that events_statements_summary_by_digest is cumulative?

  4. Is there a concern about Performance_schema_digest_lost on high-throughput instances (we run several hundred thousand QPS clusters)? What is the recommended performance_schema_digests_size setting for production workloads at this scale?

  5. Is there any plan to reintroduce a response-time distribution dashboard for MySQL 8.0/8.4 in PMM that is equivalent to the legacy “MySQL Query Response Time Details” dashboard, leveraging events_statements_histogram_by_digest?

  6. For MySQL 8.4 LTS specifically , are there any new Performance Schema enhancements or PMM 3.x features that improve per-query latency observability compared to MySQL 8.0?

    Thanks in advance.

2

Hi @pravata_dash,

Instead of answering your all 6 questions I wanted to bring your attention to something and see if that’s helpful. I think what you’re seeking is possible and you can build custom dashboards around PMM data. Have a look at this article building-query-analysis-and-insights-dashboard-in-pmm.

Towards the end there’s a ready-to-import dashboard. I believe you can extend it by creating new queries for your p95/99 investigation.

Thanks,
K