Hello Percona team,
We are seeing continuous growth of this Performance Schema mutex instance on a three-node staging PXC cluster:
wait/synch/mutex/sql/MDL_ticket_store::LOCK_ticket_store_ops
Environment
PXC: 8.4.6-6.1
Performance Schema: enabled
performance_schema_max_mutex_instances: -1 (autoscaled, before mitigation)
Monitoring: PMM / mysqld_exporter perf_schema.eventswaits collector
Related report
This appears related to the following PXC 8.0.42-33.1 report:
In that thread, the growth was reproduced and correlated with repeated short-lived monitor/health-check connections. The reporter also observed the PMM wait-summary query becoming slow and used performance_schema_max_mutex_instances=20000 as a workaround. However, the thread does not identify a public bug ID, confirmed root cause, or fixed version.
Evidence from PXC 8.4.6
Before restarting one staging node:
performance_schema.mutex_instances total: 1,048,716
LOCK_ticket_store_ops instances: 1,036,214 (98.8%)
Performance_schema_mutex_instances_lost: 16,926,214
After restart, the target count returned to approximately 200 and Performance_schema_mutex_instances_lost returned to 0.
During a four-minute observation after restart:
Connections: 131 -> 192 (+61)
LOCK_ticket_store_ops: 210 -> 271 (+61)
mutex_instances total: 2,811 -> 2,872 (+61)
The target instance count increased exactly with cumulative connections and did not return to the previous baseline after those short-lived sessions ended.
The PMM monitoring user repeatedly issued:
SELECT EVENT_NAME, COUNT_STAR, SUM_TIMER_WAIT
FROM performance_schema.events_waits_summary_global_by_event_name;
Example slow-log result:
Query_time: 1.688297
Rows_sent: 473
Rows_examined: 473
Full_scan: Yes
We confirmed that this query is issued by the PMM/mysqld_exporter perf_schema.eventswaits collector. We are reporting an observed correlation between the accumulated instances and query cost, not claiming that the global summary returns one row per mutex instance. The global summary contains one row per event name.
Additional test
We restarted a staging node with:
performance-schema-instrument='wait/synch/mutex/sql/MDL_ticket_store::LOCK_ticket_store_ops=OFF'
The instrument then reported ENABLED=NO and TIMED=NO, but its instance count still increased 1:1 with Connections. This disabled collection/timing but did not prevent instance registration.
We also inspected the PXC release-8.4.6, release-8.4.10-10, and 8.4 branches. We found initialization of m_LOCK_ticket_store_ops, but did not find a corresponding mysql_mutex_destroy() in sql/mdl.cc or sql/mdl.h. This is an observation for confirmation, not a concluded root cause.
Temporary mitigations
- Disabling only
perf_schema.eventswaitsstopped the monitoring query but removed the related wait-event metrics. - Setting
performance_schema_max_mutex_instances=20000bounded memory use, but it affects all mutex instruments. Once full,Performance_schema_mutex_instances_lostincreases and mutex observability becomes incomplete. It is not FIFO.
Questions
- Is this a known PXC defect, and is there a public issue ID?
- Is it fixed in PXC 8.4.7, 8.4.8, 8.4.10, or a later release?
- Is the absence of a visible matching
mysql_mutex_destroy()intentional, or is the mutex destroyed elsewhere? - What mitigation and
performance_schema_max_mutex_instancessizing does Percona recommend while retaining observability?
Thank you.
