- Summary
We are running PMM 3 (Percona Monitoring and Management) to monitor MySQL instances. Since December 18, 2025, we have been experiencing abnormal growth of VictoriaMetrics indexdb, which leads to high disk usage, query timeouts, and slow dashboards. We have tried several workarounds with limited success. We believe that upgrading the bundled VictoriaMetrics to v1.133.0 or later (which introduces the per-partition index, or pt-index) would address the structural cause of this issue. We would like to request that Percona consider updating the VictoriaMetrics version used in PMM accordingly.
- Current situation
- PMM version: PMM 3
- VictoriaMetrics version in use: 1.114.0 (as bundled with PMM)
- Issue: VictoriaMetrics indexdb size has been growing abnormally since December 18, 2025.
- Observed impact:
- indexdb size ratio (indexdb / (storage + indexdb)) is about 76%, whereas VictoriaMetrics documentation suggests that a ratio above 50% may indicate excessive index usage or high churn.
- Increased disk usage, query timeouts, and intermittent slowness of dashboards.
- In some cases, cardinality-related PromQL queries time out even with a 1-minute range.
- What we have tried
We have attempted the following, based on VictoriaMetrics documentation and PMM configuration options:
-
VM_STORAGE_MIN_FREE_DISK_SPACE_BYTES
We increased this value hoping to encourage more merges. It turned out this parameter acts as a safety limit (merge is blocked when free space is below this value), not a trigger for merge. So this did not help. -
Force merge API
We tried to call VictoriaMetrics’ force merge API (/internal/force_merge). In PMM 3’s architecture, VictoriaMetrics runs as an embedded single-node component, and we could not find a reliable way to invoke this API (e.g., via PMM API or from inside the container). So we could not use force merge in our environment. -
DATA_RETENTION reduction (183d to 90d to 60d)
Reducing retention did lead to indexdb merge activity and some reduction in indexdb size. However, during merge we observed intermittent gaps in metric collection. So we could not achieve “reducing indexdb without affecting collection continuity” with this approach alone.
We have also identified high cardinality as a contributing factor (e.g., many series from mysql_perf_schema_* and related MySQL metrics). We are working on reducing cardinality at the collection level (e.g., disabling some perf_schema collectors where not needed). Even with those mitigations, we believe the monolithic indexdb architecture in the current VictoriaMetrics version remains a structural limitation.
- Root cause and why we ask for a VictoriaMetrics upgrade
-
Current architecture (VictoriaMetrics 1.114.0):
A single monolithic indexDB is used for the full retention period. When old data partitions are dropped after retention, the index is not dropped with them; it is managed by a separate rotation (prev/curr/next). So indexdb can keep growing and does not shrink naturally when partitions expire. -
VictoriaMetrics v1.133.0 (released 2026-01-02):
Introduces the per-partition index (pt-index). Each partition has its own indexDB. When a partition is deleted (e.g., after retention), its indexDB is deleted with it. This reduces disk space used by the index and addresses the structural issue we are facing. -
Relevant references:
- VictoriaMetrics issue #7599 (motivation for pt-index)
- VictoriaMetrics issue #8134 (what to expect when upgrading to pt-index)
- Release note for v1.133.0: “FEATURE: vmsingle and vmstorage in VictoriaMetrics cluster : introduce per-partition index. This should reduce disk space occupied by indexDBs as they get deleted along with the corresponding partitions once those partitions become outside the retention window.”
We understand that upgrading to pt-index (v1.133.0+) may cause a one-time slowdown during rollout (as active time series are registered in the new index) and that retention shorter than one month might see increased disk usage. We are willing to accept these trade-offs if PMM can ship a VictoriaMetrics version that supports per-partition index.
- Request
We kindly request that Percona consider upgrading the VictoriaMetrics version used in PMM to v1.133.0 or later (or a subsequent version that includes the per-partition index feature), so that PMM users can benefit from:
- IndexDB being deleted together with expired partitions, reducing disk usage and avoiding unbounded index growth in high-cardinality or high-churn environments.
- A more sustainable long-term behavior for index storage without relying only on retention reduction (which in our case caused collection gaps during merge).
We would appreciate any information on:
- Whether such an upgrade is planned for a future PMM release, and if so, which version.
- Any known compatibility or testing considerations between PMM 3 and VictoriaMetrics v1.133.0+.
Thank you for your work on PMM and for considering this request.