# Pmm upgrade from 2.16 to 2.21 query analytics not starting QAN doesn't display any data **URL:** \ **Category:** PMM 2.x\ **Tags:** pmm\ **Created:** [September 20, 2021, 1:18pm UTC](https://forums.percona.com/t/pmm-upgrade-from-2-16-to-2-21-query-analytics-not-starting-qan-doesnt-display-any-data/12226 "2021-09-20T13:18:05Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![Davud\_Can](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/davud_can/32/4533_2.png) [@Davud\_Can](https://forums.percona.com/u/Davud_Can)\ **Post date:** [September 20, 2021, 1:18pm UTC](https://forums.percona.com/t/pmm-upgrade-from-2-16-to-2-21-query-analytics-not-starting-qan-doesnt-display-any-data/12226/1 "2021-09-20T13:18:05Z")
The current PMM client is a Virtual Appliance on vSphere running 2.21.0 of PMM. [root@localhost access]# /usr/sbin/percona-qan-api2 --data-retention=30 --debug --trace stdlog: qan-api2 v2.21.0. INFO[2021-09-20T14:01:17.038+00:00]qan-api2/main.go:284 main.main Log level: trace. stdlog: dsn: clickhouse://127.0.0.1:9000?database=pmm&block\_size=10000&pool\_size=2 stdlog: Migrations: Dirty database version 9. Fix and force version. [root@localhost access]# [root@localhost access]# clickhouse-client --query “select \* from pmm.schema\_migrations” 1 1 1600776273056694360 1 0 1600776273075425791 2 1 1600776273076725910 2 0 1600776273116303743 3 1 1600776273117564041 3 0 1600776273156214628 4 1 1600776273157073358 4 0 1600776273181985508 5 1 1600776273182842458 5 0 1600776273193594158 6 1 1600776273194365739 6 0 1600776273203387813 7 1 1620132085016538511 7 0 1620132085071669360 8 1 1620132085072878986 8 0 1620132085173742781 9 1 1631874203339757424 [root@localhost access]# thre result of the supervisorctl command is : [root@localhost ~]# supervisorctl status alertmanager RUNNING pid 1105, uptime 1:14:25 clickhouse RUNNING pid 1099, uptime 1:14:25 cron RUNNING pid 1102, uptime 1:14:25 dashboard-upgrade EXITED Sep 20 11:44 AM dbaas-controller STOPPED Not started grafana RUNNING pid 1100, uptime 1:14:25 nginx RUNNING pid 1101, uptime 1:14:25 pmm-agent RUNNING pid 1109, uptime 1:14:25 pmm-managed RUNNING pid 1174, uptime 1:14:19 pmm-update-perform STOPPED Not started postgresql RUNNING pid 1098, uptime 1:14:25 prometheus STOPPED Not started qan-api2 STARTING victoriametrics RUNNING pid 1103, uptime 1:14:25 vmalert RUNNING pid 1104, uptime 1:14:25 [root@localhost ~]# ---
**Author:** ![Michael\_Coburn](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/michael_coburn/32/18_2.png) [@Michael\_Coburn](https://forums.percona.com/u/Michael_Coburn)\ **Post date:** [September 20, 2021, 2:16pm UTC](https://forums.percona.com/t/pmm-upgrade-from-2-16-to-2-21-query-analytics-not-starting-qan-doesnt-display-any-data/12226/2 "2021-09-20T14:16:15Z")
Hi @Davud_Can , welcome to the Percona forums! Could you please show us the logs from the file `/srv/logs/qan-api2.log` so that we can see where the application is failing to start? ---
**Author:** ![Davud\_Can](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/davud_can/32/4533_2.png) [@Davud\_Can](https://forums.percona.com/u/Davud_Can)\ **Post date:** [September 20, 2021, 3:51pm UTC](https://forums.percona.com/t/pmm-upgrade-from-2-16-to-2-21-query-analytics-not-starting-qan-doesnt-display-any-data/12226/3 "2021-09-20T15:51:45Z")
Hi @Michael_Coburn Thank you for your quick return. I have attached the log file you requested. Regards[qan-api2.log](https://forums.percona.com/uploads/short-url/hsMnSDFuJet9yMpfrOL9b3Y0lKO.log) (65.9 KB) ---
**Author:** ![Michael\_Coburn](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/michael_coburn/32/18_2.png) [@Michael\_Coburn](https://forums.percona.com/u/Michael_Coburn)\ **Post date:** [September 20, 2021, 6:25pm UTC](https://forums.percona.com/t/pmm-upgrade-from-2-16-to-2-21-query-analytics-not-starting-qan-doesnt-display-any-data/12226/4 "2021-09-20T18:25:01Z")
Hi @Davud_Can , thanks for sharing the logs. As you will notice, there is a recurring failure every 2 seconds ```auto e[36mINFOe[0m[2021-09-20T15:46:10.212+00:00] Log level: info. stdlog: dsn: clickhouse://127.0.0.1:9000?database=pmm&block_size=10000&pool_size=2 stdlog: Migrations: Dirty database version 9. Fix and force version. stdlog: qan-api2 v2.21.0. ``` I haven’t seen this error before, and I am escalating it to the PMM Engineering team. It seems there is an issue with the ClickHouse schema. ---
**Author:** ![Davud\_Can](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/davud_can/32/4533_2.png) [@Davud\_Can](https://forums.percona.com/u/Davud_Can)\ **Post date:** [September 20, 2021, 7:03pm UTC](https://forums.percona.com/t/pmm-upgrade-from-2-16-to-2-21-query-analytics-not-starting-qan-doesnt-display-any-data/12226/5 "2021-09-20T19:03:12Z")
Thanks a lot for the information. I will wait for news from you. ---
**Author:** ![nurlan](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/nurlan/32/1523_2.png) [@nurlan](https://forums.percona.com/u/nurlan)\ **Post date:** [September 22, 2021, 9:27am UTC](https://forums.percona.com/t/pmm-upgrade-from-2-16-to-2-21-query-analytics-not-starting-qan-doesnt-display-any-data/12226/6 "2021-09-22T09:27:25Z")
Hi @Davud_Can, Could you, please connect to clichouse using clickhouse-client installed in our docker image and share results of this command. - describe table pmm.metrics; Thank you ---
**Author:** ![Davud\_Can](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/davud_can/32/4533_2.png) [@Davud\_Can](https://forums.percona.com/u/Davud_Can)\ **Post date:** [September 22, 2021, 11:34am UTC](https://forums.percona.com/t/pmm-upgrade-from-2-16-to-2-21-query-analytics-not-starting-qan-doesnt-display-any-data/12226/7 "2021-09-22T11:34:07Z")
Hi @nurlan I am attaching the output you requested as an attachment. Regards,[desc\_pmm\_metrics.txt](https://forums.percona.com/uploads/short-url/utSIqiphYowbsjKM1fbTyXimgOm.txt) (12.4 KB) ---
**Author:** ![Davud\_Can](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/davud_can/32/4533_2.png) [@Davud\_Can](https://forums.percona.com/u/Davud_Can)\ **Post date:** [October 5, 2021, 10:45am UTC](https://forums.percona.com/t/pmm-upgrade-from-2-16-to-2-21-query-analytics-not-starting-qan-doesnt-display-any-data/12226/9 "2021-10-05T10:45:13Z")
Hello @nurlan Is there any news on the subject? Regards, ---
**Author:** ![nurlan](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/nurlan/32/1523_2.png) [@nurlan](https://forums.percona.com/u/nurlan)\ **Post date:** [October 5, 2021, 8:34pm UTC](https://forums.percona.com/t/pmm-upgrade-from-2-16-to-2-21-query-analytics-not-starting-qan-doesnt-display-any-data/12226/10 "2021-10-05T20:34:36Z")
Hi @Davud_Can We couldn’t reproduce this behavior, but we have workaround for you. could you do these manual steps in your clickhouse? 1. log in into clickhouse. 2. go to `pmm` db using `use pmm;` 3. execute ```auto ALTER TABLE metrics ADD COLUMN `m_plans_calls_cnt` Float32, ADD COLUMN `m_plans_calls_sum` Float32 COMMENT 'Total number of planned calls', ADD COLUMN `m_wal_records_cnt` Float32, ADD COLUMN `m_wal_records_sum` Float32 COMMENT 'Total number of WAL (Write-ahead logging) records', ADD COLUMN `m_wal_fpi_cnt` Float32, ADD COLUMN `m_wal_fpi_sum` Float32 COMMENT 'Total number of FPI (full page images) in WAL (Write-ahead logging) records'; ``` 1. execute `insert into schema_migrations (version, dirty, sequence) values(9, 0, toUnixTimestamp(NOW())*1000000000);` Thank you ---
**Author:** ![Davud\_Can](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/davud_can/32/4533_2.png) [@Davud\_Can](https://forums.percona.com/u/Davud_Can)\ **Post date:** [October 7, 2021, 8:56am UTC](https://forums.percona.com/t/pmm-upgrade-from-2-16-to-2-21-query-analytics-not-starting-qan-doesnt-display-any-data/12226/11 "2021-10-07T08:56:21Z")
Hi @nurlan Thank you for your precious help, these solution has worked for us and solved our problem. Regards,