# Percona PMM query analytics is broken **URL:** \ **Category:** Percona Monitoring and Management (PMM)\ **Tags:** grafana, pmm, postgresql\ **Created:** [August 26, 2025, 7:17am UTC](https://forums.percona.com/t/percona-pmm-query-analytics-is-broken/39201 "2025-08-26T07:17:26Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![aniruddha.deshpande](https://avatars.discourse-cdn.com/v4/letter/a/ecb155/32.png) [@aniruddha.deshpande](https://forums.percona.com/u/aniruddha.deshpande)\ **Post date:** [August 26, 2025, 7:17am UTC](https://forums.percona.com/t/percona-pmm-query-analytics-is-broken/39201/1 "2025-08-26T07:17:26Z")
## Description: We are using PMM to monitor our databases. Was using version PMM 3.2.0 & even upgraded to latest version 3.3.1. We see that we are getting below error when we try to look for QAN. And this is persistent across old version and new version of PMM. ![pmm-error](https://us1.discourse-cdn.com/flex019/uploads/percona1/original/3X/5/9/59131fded49ba1195279472dfba25fdb77989427.jpeg) ## Steps to Reproduce: [Step-by-step instructions on how to reproduce the issue, including any specific settings or configurations] ## Version: 3.3.1 ## ---
**Author:** ![aniruddha.deshpande](https://avatars.discourse-cdn.com/v4/letter/a/ecb155/32.png) [@aniruddha.deshpande](https://forums.percona.com/u/aniruddha.deshpande)\ **Post date:** [August 26, 2025, 7:24am UTC](https://forums.percona.com/t/percona-pmm-query-analytics-is-broken/39201/2 "2025-08-26T07:24:14Z")
To be clear, this error we got directly even clicking on PMM QAN. We are not selecting anything. ---
**Author:** ![aniruddha.deshpande](https://avatars.discourse-cdn.com/v4/letter/a/ecb155/32.png) [@aniruddha.deshpande](https://forums.percona.com/u/aniruddha.deshpande)\ **Post date:** [August 26, 2025, 9:19am UTC](https://forums.percona.com/t/percona-pmm-query-analytics-is-broken/39201/3 "2025-08-26T09:19:43Z")
More information - In the qan-api2.log logs, we see below error > time=“2025-08-26T06:12:22.254+00:00” level=info msg=“Log level: info.” > time=“2025-08-26T06:12:22.254+00:00” level=info msg=“DSN: clickhouse://127.0.0.1:9000?database=pmm&block\_size=10000&pool\_size=2” component=main > stdlog: dsn: clickhouse://127.0.0.1:9000?database=pmm&block\_size=10000&pool\_size=2 > stdlog: Migrations: no migration found for version 20: read down for version 20 migrations/sql: file does not exist > stdlog: qan-api2 v2.44.0. > time=“2025-08-26T06:12:41.620+00:00” level=info msg=“Log level: info.” > time=“2025-08-26T06:12:41.620+00:00” level=info msg=“DSN: clickhouse://127.0.0.1:9000?database=pmm&block\_size=10000&pool\_size=2” component=main > stdlog: dsn: clickhouse://127.0.0.1:9000?database=pmm&block\_size=10000&pool\_size=2 > stdlog: Migrations: no migration found for version 20: read down for version 20 migrations/sql: file does not exist > stdlog: qan-api2 v2.44.0. > time=“2025-08-26T06:13:01.669+00:00” level=info msg=“Log level: info.” > time=“2025-08-26T06:13:01.669+00:00” level=info msg=“DSN: clickhouse://127.0.0.1:9000?database=pmm&block\_size=10000&pool\_size=2” component=main > stdlog: dsn: clickhouse://127.0.0.1:9000?database=pmm&block\_size=10000&pool\_size=2 > stdlog: Migrations: no migration found for version 20: read down for version 20 migrations/sql: file does not exist > stdlog: qan-api2 v2.44.0. > time=“2025-08-26T06:13:22.805+00:00” level=info msg=“Log level: info.” > time=“2025-08-26T06:13:22.805+00:00” level=info msg=“DSN: clickhouse://127.0.0.1:9000?database=pmm&block\_size=10000&pool\_size=2” component=main > stdlog: dsn: clickhouse://127.0.0.1:9000?database=pmm&block\_size=10000&pool\_size=2 > stdlog: Migrations: no migration found for version 20: read down for version 20 migrations/sql: file does not exist > stdlog: qan-api2 v2.44.0. > time=“2025-08-26T06:13:44.968+00:00” level=info msg=“Log level: info.” > time=“2025-08-26T06:13:44.968+00:00” level=info msg=“DSN: clickhouse://127.0.0.1:9000?database=pmm&block\_size=10000&pool\_size=2” component=main > stdlog: dsn: clickhouse://127.0.0.1:9000?database=pmm&block\_size=10000&pool\_size=2 > stdlog: Migrations: no migration found for version 20: read down for version 20 migrations/sql: file does not exist > stdlog: qan-api2 v2.44.0. > time=“2025-08-26T06:14:08.017+00:00” level=info msg=“Log level: info.” > time=“2025-08-26T06:14:08.017+00:00” level=info msg=“DSN: clickhouse://127.0.0.1:9000?database=pmm&block\_size=10000&pool\_size=2” component=main > stdlog: dsn: clickhouse://127.0.0.1:9000?database=pmm&block\_size=10000&pool\_size=2 > stdlog: Migrations: no migration found for version 20: read down for version 20 migrations/sql: file does not exist > stdlog: qan-api2 v2.44.0. > time=“2025-08-26T06:14:32.247+00:00” level=info msg=“Log level: info.” > time=“2025-08-26T06:14:32.247+00:00” level=info msg=“DSN: clickhouse://127.0.0.1:9000?database=pmm&block\_size=10000&pool\_size=2” component=main > stdlog: dsn: clickhouse://127.0.0.1:9000?database=pmm&block\_size=10000&pool\_size=2 > stdlog: Migrations: no migration found for version 20: read down for version 20 migrations/sql: file does not exist > stdlog: qan-api2 v2.44.0. > time=“2025-08-26T06:14:57.363+00:00” level=info msg=“Log level: info.” > time=“2025-08-26T06:14:57.363+00:00” level=info msg=“DSN: clickhouse://127.0.0.1:9000?database=pmm&block\_size=10000&pool\_size=2” component=main > stdlog: dsn: clickhouse://127.0.0.1:9000?database=pmm&block\_size=10000&pool\_size=2 > stdlog: Migrations: no migration found for version 20: read down for version 20 migrations/sql: file does not exist > stdlog: qan-api2 v3.2.0. > time=“2025-08-26T06:15:36.466+00:00” level=info msg=“Log level: info.” > time=“2025-08-26T06:15:36.466+00:00” level=info msg=“DSN: clickhouse://default:xxxxx@127.0.0.1:9000/pmm” component=main > stdlog: Connection: dial tcp 127.0.0.1:9000: connect: connection refused > stdlog: qan-api2 v3.2.0. > time=“2025-08-26T06:15:37.718+00:00” level=info msg=“Log level: info.” > time=“2025-08-26T06:15:37.718+00:00” level=info msg=“DSN: clickhouse://default:xxxxx@127.0.0.1:9000/pmm” component=main > stdlog: Connection: dial tcp 127.0.0.1:9000: connect: connection refused > stdlog: qan-api2 v3.2.0. > time=“2025-08-26T06:15:39.760+00:00” level=info msg=“Log level: info.” > time=“2025-08-26T06:15:39.760+00:00” level=info msg=“DSN: clickhouse://default:xxxxx@127.0.0.1:9000/pmm” component=main > stdlog: Connection: dial tcp 127.0.0.1:9000: connect: connection refused > stdlog: qan-api2 v3.2.0. > time=“2025-08-26T06:15:42.925+00:00” level=info msg=“Log level: info.” > time=“2025-08-26T06:15:42.925+00:00” level=info msg=“DSN: clickhouse://default:xxxxx@127.0.0.1:9000/pmm” component=main > stdlog: Connection: dial tcp 127.0.0.1:9000: connect: connection refused > stdlog: qan-api2 v3.2.0. > time=“2025-08-26T06:15:46.971+00:00” level=info msg=“Log level: info.” > time=“2025-08-26T06:15:46.971+00:00” level=info msg=“DSN: clickhouse://default:xxxxx@127.0.0.1:9000/pmm” component=main > stdlog: Connection: dial tcp 127.0.0.1:9000: connect: connection refused > stdlog: qan-api2 v3.2.0. > time=“2025-08-26T06:15:52.002+00:00” level=info msg=“Log level: info.” > time=“2025-08-26T06:15:52.002+00:00” level=info msg=“DSN: clickhouse://default:xxxxx@127.0.0.1:9000/pmm” component=main > stdlog: Connection: dial tcp 127.0.0.1:9000: connect: connection refused > stdlog: qan-api2 v3.2.0. > time=“2025-08-26T06:15:58.116+00:00” level=info msg=“Log level: info.” > time=“2025-08-26T06:15:58.116+00:00” level=info msg=“DSN: clickhouse://default:xxxxx@127.0.0.1:9000/pmm” component=main > stdlog: Connection: dial tcp 127.0.0.1:9000: connect: connection refused > stdlog: qan-api2 v3.2.0. ---
**Author:** ![aniruddha.deshpande](https://avatars.discourse-cdn.com/v4/letter/a/ecb155/32.png) [@aniruddha.deshpande](https://forums.percona.com/u/aniruddha.deshpande)\ **Post date:** [August 26, 2025, 9:37am UTC](https://forums.percona.com/t/percona-pmm-query-analytics-is-broken/39201/4 "2025-08-26T09:37:22Z")
Update : During the investigation, we saw that the click house was not running inside docker. and upon checking logs - ```auto 2025.08.26 09:22:43.008963 [70154] {} Application: Caught exception while loading metadata: Code: 76. DB::ErrnoException: Cannot open file /srv/clickhouse/store/3c8/3c856d25-599f-4ecb-a263-2aedc01ec43e/asynchronous_metric_log.sql, errno: 13, strerror: Permission denied: Cannot parse definition from metadata file /srv/clickhouse/store/3c8/3c856d25-599f-4ecb-a263-2aedc01ec43e/asynchronous_metric_log.sql. (CANNOT_OPEN_FILE), Stack trace (when copying this message, always include the lines below):2025.08.26 09:22:43.924935 [70154] {} Application: Code: 76. DB::ErrnoException: Cannot open file /srv/clickhouse/store/3c8/3c856d25-599f-4ecb-a263-2aedc01ec43e/asynchronous_metric_log.sql, errno: 13, strerror: Permission denied: Cannot parse definition from metadata file /srv/clickhouse/store/3c8/3c856d25-599f-4ecb-a263-2aedc01ec43e/asynchronous_metric_log.sql. (CANNOT_OPEN_FILE), Stack trace (when copying this message, always include the lines below):2025.08.26 09:22:43.925044 [70154] {} Application: DB::ErrnoException: Cannot open file /srv/clickhouse/store/3c8/3c856d25-599f-4ecb-a263-2aedc01ec43e/asynchronous_metric_log.sql, errno: 13, strerror: Permission denied: Cannot parse definition from metadata file /srv/clickhouse/store/3c8/3c856d25-599f-4ecb-a263-2aedc01ec43e/asynchronous_metric_log.sql ``` and fix we applied - chown -R pmm:pmm /srv/clickhouse/ ---
**Author:** ![aniruddha.deshpande](https://avatars.discourse-cdn.com/v4/letter/a/ecb155/32.png) [@aniruddha.deshpande](https://forums.percona.com/u/aniruddha.deshpande)\ **Post date:** [August 26, 2025, 11:37am UTC](https://forums.percona.com/t/percona-pmm-query-analytics-is-broken/39201/5 "2025-08-26T11:37:45Z")
can someone from percona help us on how this has had happened? ---
**Author:** ![ademidoff](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/ademidoff/32/14234_2.png) [@ademidoff](https://forums.percona.com/u/ademidoff)\ **Post date:** [August 26, 2025, 12:59pm UTC](https://forums.percona.com/t/percona-pmm-query-analytics-is-broken/39201/6 "2025-08-26T12:59:02Z")
Hi What is the output of `supervisorctl status`? Is ClickHouse running after the fix you applied? If not, can you try to bring it up via `supervisorctl start clickhouse` command? Cheers ---
**Author:** ![aniruddha.deshpande](https://avatars.discourse-cdn.com/v4/letter/a/ecb155/32.png) [@aniruddha.deshpande](https://forums.percona.com/u/aniruddha.deshpande)\ **Post date:** [August 26, 2025, 1:03pm UTC](https://forums.percona.com/t/percona-pmm-query-analytics-is-broken/39201/7 "2025-08-26T13:03:50Z")
> [@ademidoff](#): > > supervisorctl status Hi Alex, Thank you for reply. After applying the fix, the clickhouse was running again and I was able to see QAN. Current output - [root@eb6e746ea3ba opt] # supervisorctl status clickhouse RUNNING pid 70889, uptime 3:36:23 grafana RUNNING pid 429, uptime 6:36:15 nginx RUNNING pid 16, uptime 6:36:26 nomad-server STOPPED Not started pmm-agent RUNNING pid 594, uptime 6:36:01 pmm-init EXITED Aug 26 06:23 AM pmm-managed RUNNING pid 25, uptime 6:36:26 postgresql RUNNING pid 13, uptime 6:36:26 qan-api2 RUNNING pid 70764, uptime 3:36:49 victoriametrics RUNNING pid 581, uptime 6:36:02 vmalert RUNNING pid 18, uptime 6:36:26 vmproxy RUNNING pid 19, uptime 6:36:26 But the problem was, we had an incident this morning and wanted to look in to QAN when we realized that QAN is no more working. No one has modified or touch this docker since last upgrade. The only that we changed after we upgrade last time was, using our own SSL certificates. I have logs from PMM that i can share with you but can do that privately over email or link. (due to obvious reasons) ---
**Author:** ![ademidoff](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/ademidoff/32/14234_2.png) [@ademidoff](https://forums.percona.com/u/ademidoff)\ **Post date:** [August 26, 2025, 1:59pm UTC](https://forums.percona.com/t/percona-pmm-query-analytics-is-broken/39201/8 "2025-08-26T13:59:54Z")
Glad to hear the issue was resolved! I suggest you check the container health or run `supervisorctl status` after each upgrade. No, we will never ask to share the logs privately 🙂 ---
**Author:** ![aniruddha.deshpande](https://avatars.discourse-cdn.com/v4/letter/a/ecb155/32.png) [@aniruddha.deshpande](https://forums.percona.com/u/aniruddha.deshpande)\ **Post date:** [August 26, 2025, 2:38pm UTC](https://forums.percona.com/t/percona-pmm-query-analytics-is-broken/39201/9 "2025-08-26T14:38:43Z")
> [@ademidoff](#): > > supervisorctl status thank you for the answer. But can this not be done by upgrade process itself? or documented anywhere ? (sorry if its done already) . We used upgrade from UI itself and if it says success, probably lot of people will trust it. ---
**Author:** ![ademidoff](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/ademidoff/32/14234_2.png) [@ademidoff](https://forums.percona.com/u/ademidoff)\ **Post date:** [August 26, 2025, 3:12pm UTC](https://forums.percona.com/t/percona-pmm-query-analytics-is-broken/39201/10 "2025-08-26T15:12:50Z")
I do not rule out there is a bug. However, we don’t capture anything like that in our automated tests. And I don’t even have a good clue as to what could have happened ☹