PMM upgrade from 2.44.1 to 3.2.0 - Migrations: Dirty database version 18

Percona Monitoring and Management (PMM) PMM 3.x
1

Hello there,

As we plan to upgrade MySQL to version 8.4, we also need to upgrade PMM as MySQL 8.4 has been supported since PMM 3.2.
As described here Migrate PMM 2 to PMM 3 - Percona Monitoring and Management, we first updated PMM from 2.43.1 to 2.44.1 (server and clients) and everything works fine.
Now when we try to upgrade PMM server to version 3.2.0, the container has a healthy status but when we look at the QAN log we see the following messages

time="2025-07-07T12:43:11.585+00:00" level=info msg="Log level: info."
time="2025-07-07T12:43:11.585+00:00" level=info msg="DSN: clickhouse://default:xxxxx@127.0.0.1:9000/pmm" component=main
stdlog: Connection: dial tcp 127.0.0.1:9000: connect: connection refused
stdlog: qan-api2 v3.2.0.
time="2025-07-07T12:43:16.642+00:00" level=info msg="Log level: info."
time="2025-07-07T12:43:16.642+00:00" level=info msg="DSN: clickhouse://default:xxxxx@127.0.0.1:9000/pmm" component=main
stdlog: Migrations: Dirty database version 18. Fix and force version.
stdlog: qan-api2 v3.2.0.

If we query schema_migrations table from clickhouse we get this :

┌─version─┬─dirty─┬────────────sequence─┐
│       1 │     1 │ 1666304163282803150 │
│       1 │     0 │ 1666304163307409112 │
│       2 │     1 │ 1666304163308201705 │
│       2 │     0 │ 1666304163319120756 │
│       3 │     1 │ 1666304163320110562 │
│       3 │     0 │ 1666304163332385053 │
...
│      16 │     1 │ 1685547240793994367 │
│      16 │     0 │ 1685547240813912294 │
│      17 │     1 │ 1732630983714570596 │
│      17 │     0 │ 1732631024675773683 │
│      18 │     1 │ 1751881578895683603 │
└─────────┴───────┴─────────────────────┘

We have decided to compare with an installation from scratch and in this case the table contains


┌─version─┬─dirty─┬────────────sequence─┐
│       1 │     1 │ 1751889867602556746 │
│       1 │     0 │ 1751889867642858002 │
│       2 │     1 │ 1751889867651773260 │
│       2 │     0 │ 1751889867680239360 │
│       3 │     1 │ 1751889867687664041 │
│       3 │     0 │ 1751889867708692299 │
...
│      16 │     1 │ 1751889868120175656 │
│      16 │     0 │ 1751889868145455301 │
│      17 │     1 │ 1751889868153159388 │
│      17 │     0 │ 1751889868187090026 │
│      18 │     1 │ 1751889868195827440 │
│      18 │     0 │ 1751889868220447379 │
│      19 │     1 │ 1751889868235411643 │
│      19 │     0 │ 1751889868255468951 │
│      20 │     1 │ 1751889868263304656 │
│      20 │     0 │ 1751889868293712090 │
└─────────┴───────┴─────────────────────┘

The clickhouse database does not seem to have been updated correctly during migration. We encountered this problem once with a previous version and ran the migration script manually.
We’ve looked at the Gihtub repo : pmm/qan-api2/migrations/sql at main · percona/pmm · GitHub but there is no migration script for versions 18 and above.

Can you help us please?

2

Hi, the migration files are in the v3 branch and there, you’d see the latest scripts: pmm/qan-api2/migrations/sql at v3 · percona/pmm · GitHub

3

Hi @Michael_Okoko ,

Thanks for your help !

After analyzing the migration script for the 18th version, the script was applied and the clickhouse metrics table had the correct structure. We have just run the following query :

insert into schema_migrations (version, dirty, sequence) values(18, 0, toUnixTimestamp(NOW())*1000000000);

And the migration was then completed automatically.

On another PMM server we also realized the migration from 2.44.1 to 3.2.0 but this time everything worked correctly (no manual action was required).
Finally we don’t know why the migration partially failed the first time.