PMM not starting after upgrade to 3.9.1

Percona Monitoring and Management (PMM) PMM 3.x
1

Hello Percona team,

after performing an upgrade from PMM 3.7.1 to 3.9.1 (Docker) the upgrated container is not coming up with no apparent reasons.

Running a “docker logs ” I get this:

Running as UID 1000
Creating nginx temp directories...
Generating self-signed certificates for nginx...
Checking nginx configuration...
2026/08/27 08:40:28 [warn] 20#20: the "listen ... http2" directive is deprecated, use the "http2" directive instead in /etc/nginx/conf.d/pmm.conf:45
nginx: [warn] the "listen ... http2" directive is deprecated, use the "http2" directive instead in /etc/nginx/conf.d/pmm.conf:45
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
time="2026-08-27T08:40:28.432+00:00" level=info msg="pmm.ini configuration has been updated."
2026-08-27 08:40:28,617 INFO Included extra file "/etc/supervisord.d/grafana.ini" during parsing
2026-08-27 08:40:28,617 INFO Included extra file "/etc/supervisord.d/pmm.ini" during parsing
2026-08-27 08:40:28,617 INFO Included extra file "/etc/supervisord.d/qan-api2.ini" during parsing
2026-08-27 08:40:28,617 INFO Included extra file "/etc/supervisord.d/supervisord.ini" during parsing
2026-08-27 08:40:28,617 INFO Included extra file "/etc/supervisord.d/victoriametrics.ini" during parsing
2026-08-27 08:40:28,617 INFO Included extra file "/etc/supervisord.d/vmalert.ini" during parsing
2026-08-27 08:40:28,617 INFO Included extra file "/etc/supervisord.d/vmproxy.ini" during parsing
2026-08-27 08:40:28,620 INFO RPC interface 'supervisor' initialized
2026-08-27 08:40:28,620 INFO supervisord started with pid 1
2026-08-27 08:40:29,623 INFO spawned: 'pmm-init' with pid 26
2026-08-27 08:40:29,625 INFO spawned: 'postgresql' with pid 27
2026-08-27 08:40:29,627 INFO spawned: 'clickhouse' with pid 28
2026-08-27 08:40:29,629 INFO spawned: 'grafana' with pid 29
2026-08-27 08:40:29,630 INFO spawned: 'nginx' with pid 30
2026-08-27 08:40:29,633 INFO spawned: 'victoriametrics' with pid 33
2026-08-27 08:40:29,635 INFO spawned: 'vmalert' with pid 34
2026-08-27 08:40:29,637 INFO spawned: 'vmproxy' with pid 35
2026-08-27 08:40:29,639 INFO spawned: 'qan-api2' with pid 37
2026-08-27 08:40:29,640 INFO spawned: 'pmm-managed' with pid 45
2026-08-27 08:40:29,653 INFO exited: qan-api2 (exit status 1; not expected)
2026-08-27 08:40:29,747 INFO exited: postgresql (exit status 1; not expected)
2026-08-27 08:40:29,827 INFO exited: grafana (exit status 1; not expected)
2026-08-27 08:40:30,629 INFO success: pmm-init entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2026-08-27 08:40:30,629 INFO success: clickhouse entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2026-08-27 08:40:30,630 INFO success: nginx entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2026-08-27 08:40:30,669 INFO success: victoriametrics entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2026-08-27 08:40:30,669 INFO success: vmalert entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2026-08-27 08:40:30,670 INFO success: vmproxy entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2026-08-27 08:40:30,671 INFO spawned: 'qan-api2' with pid 823
2026-08-27 08:40:30,672 INFO success: pmm-managed entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2026-08-27 08:40:30,749 INFO spawned: 'postgresql' with pid 837
2026-08-27 08:40:30,834 INFO spawned: 'grafana' with pid 843
2026-08-27 08:40:30,890 INFO exited: postgresql (exit status 1; not expected)
2026-08-27 08:40:31,050 INFO exited: grafana (exit status 1; not expected)
2026-08-27 08:40:31,670 INFO success: qan-api2 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2026-08-27 08:40:32,896 INFO spawned: 'postgresql' with pid 948
2026-08-27 08:40:33,041 INFO exited: postgresql (exit status 1; not expected)
2026-08-27 08:40:33,079 INFO spawned: 'grafana' with pid 961
2026-08-27 08:40:33,310 INFO exited: grafana (exit status 1; not expected)
2026-08-27 08:40:33,390 INFO exited: pmm-init (exit status 0; expected)
2026-08-27 08:40:36,064 INFO spawned: 'postgresql' with pid 978
2026-08-27 08:40:36,136 INFO exited: postgresql (exit status 1; not expected)
2026-08-27 08:40:36,358 INFO spawned: 'grafana' with pid 983
2026-08-27 08:40:36,481 INFO exited: grafana (exit status 1; not expected)
2026-08-27 08:40:40,185 INFO spawned: 'postgresql' with pid 1007
2026-08-27 08:40:40,260 INFO exited: postgresql (exit status 1; not expected)
2026-08-27 08:40:40,486 INFO spawned: 'grafana' with pid 1008
2026-08-27 08:40:40,633 INFO exited: grafana (exit status 1; not expected)
2026-08-27 08:40:45,288 INFO spawned: 'postgresql' with pid 1039
2026-08-27 08:40:45,362 INFO exited: postgresql (exit status 1; not expected)
2026-08-27 08:40:45,637 INFO spawned: 'grafana' with pid 1040
2026-08-27 08:40:45,763 INFO exited: grafana (exit status 1; not expected)
2026-08-27 08:40:51,368 INFO spawned: 'postgresql' with pid 1065
2026-08-27 08:40:51,445 INFO exited: postgresql (exit status 1; not expected)
2026-08-27 08:40:51,770 INFO spawned: 'grafana' with pid 1066
2026-08-27 08:40:51,893 INFO exited: grafana (exit status 1; not expected)
2026-08-27 08:40:58,448 INFO spawned: 'postgresql' with pid 1099
2026-08-27 08:40:58,525 INFO exited: postgresql (exit status 1; not expected)
2026-08-27 08:40:58,896 INFO spawned: 'grafana' with pid 1100
2026-08-27 08:40:59,028 INFO exited: grafana (exit status 1; not expected)
2026-08-27 08:41:06,533 INFO spawned: 'postgresql' with pid 1131
2026-08-27 08:41:06,608 INFO exited: postgresql (exit status 1; not expected)
2026-08-27 08:41:07,030 INFO spawned: 'grafana' with pid 1132
2026-08-27 08:41:07,155 INFO exited: grafana (exit status 1; not expected)
2026-08-27 08:41:15,613 INFO spawned: 'postgresql' with pid 1164
2026-08-27 08:41:15,730 INFO exited: postgresql (exit status 1; not expected)
2026-08-27 08:41:16,158 INFO spawned: 'grafana' with pid 1165
2026-08-27 08:41:16,286 INFO exited: grafana (exit status 1; not expected)
2026-08-27 08:41:25,789 INFO spawned: 'postgresql' with pid 1203
2026-08-27 08:41:25,862 INFO exited: postgresql (exit status 1; not expected)
2026-08-27 08:41:25,872 INFO gave up: postgresql entered FATAL state, too many start retries too quickly
2026-08-27 08:41:26,315 INFO spawned: 'grafana' with pid 1204
2026-08-27 08:41:26,437 INFO exited: grafana (exit status 1; not expected)
2026-08-27 08:41:26,451 INFO gave up: grafana entered FATAL state, too many start retries too quickly

Basically, there are no obvious reasons for this issue.

Would you be able to give us some light on this issue, if possible?

Thanks.

2

Some additional “odd” errors that I could observe in some “/srv/logs” files:

###
### /srv/logs/qan-api2.log
###
me="2026-08-27T08:19:19.833+00:00" level=info msg="Log level: info."
time="2026-08-27T08:19:19.833+00:00" level=info msg="DSN: clickhouse://default:xxxxx@127.0.0.1:9000/pmm" component=main
time="2026-08-27T08:19:19.833+00:00" level=info msg="New connection with DSN: clickhouse://default:xxxxx@127.0.0.1:9000/pmm" component=db
time="2026-08-27T08:19:19.833+00:00" level=error msg="Error connecting to ClickHouse: dial tcp 127.0.0.1:9000: connect: connection refused" component=db
time="2026-08-27T08:19:19.833+00:00" level=fatal msg="Connection: dial tcp 127.0.0.1:9000: connect: connection refused" component=db


###
### /srv/logs/postgresql14.log
###
2026-08-27 08:02:13.937 UTC [47] LOG:  starting PostgreSQL 14.24 - Percona Distribution on x86_64-pc-linux-gnu, compiled by gcc (GCC) 14.2.1 20250110 (Red Hat 14.2.1-13), 64-bit
2026-08-27 08:02:13.942 UTC [47] LOG:  listening on IPv6 address "::1", port 5432
2026-08-27 08:02:13.942 UTC [47] LOG:  listening on IPv4 address "127.0.0.1", port 5432
2026-08-27 08:02:13.951 UTC [47] LOG:  listening on Unix socket "/run/postgresql/.s.PGSQL.5432"
2026-08-27 08:02:13.959 UTC [47] LOG:  listening on Unix socket "/tmp/.s.PGSQL.5432"
2026-08-27 08:02:13.966 UTC [80] LOG:  database system was interrupted; last known up at 2026-08-27 07:37:19 UTC
2026-08-27 08:02:13.992 UTC [80] LOG:  database system was not properly shut down; automatic recovery in progress
2026-08-27 08:02:14.017 UTC [80] LOG:  redo starts at 69/B9E34870
2026-08-27 08:02:14.053 UTC [80] LOG:  invalid record length at 69/BA2E69B0: wanted 24, got 0
2026-08-27 08:02:14.053 UTC [80] LOG:  redo done at 69/BA2E6988 system usage: CPU: user: 0.01 s, system: 0.01 s, elapsed: 0.03 s
2026-08-27 08:02:14.238 UTC [47] LOG:  database system is ready to accept connections
2026-08-27 08:02:22.195 UTC [1172] ERROR:  relation "unifiedstorage_migration_log" does not exist at character 22
2026-08-27 08:02:22.195 UTC [1172] STATEMENT:  SELECT count(*) FROM "unifiedstorage_migration_log" WHERE (migration_id = $1)
2026-08-27 08:09:24.380 UTC [27] FATAL:  lock file "postmaster.pid" already exists
2026-08-27 08:09:24.380 UTC [27] HINT:  Is another postmaster (PID 47) running in data directory "/srv/postgres14"?
2026-08-27 08:09:25.410 UTC [130] FATAL:  lock file "postmaster.pid" already exists
2026-08-27 08:09:25.410 UTC [130] HINT:  Is another postmaster (PID 47) running in data directory "/srv/postgres14"?
2026-08-27 08:09:27.444 UTC [923] FATAL:  lock file "postmaster.pid" already exists
2026-08-27 08:09:27.444 UTC [923] HINT:  Is another postmaster (PID 47) running in data directory "/srv/postgres14"?


###
### /srv/logs/pmm-managed.log
###
time="2026-08-27T08:17:19.799+00:00" level=info msg="Migrating database..." component=migration
time="2026-08-27T08:17:19.799+00:00" level=warning msg="Failed to migrate database: dial tcp 127.0.0.1:5432: connect: connection refused." component=migration
time="2026-08-27T08:17:20.799+00:00" level=fatal msg="Could not migrate DB: timeout" component=migration
time="2026-08-27T08:17:20.896+00:00" level=info msg="Log level: info."
time="2026-08-27T08:17:20.897+00:00" level=info msg="Using default metrics config" component=config
time="2026-08-27T08:17:20.900+00:00" level=info msg="Using Percona Platform address: https://check.percona.com."
time="2026-08-27T08:17:20.900+00:00" level=info msg="Migrating database..." component=migration

Not sure if these are related, are causing or are just victims of the issue.

3

This is sorted now.

The issue was related to “pmm.conf” file, in which I used from the previous deployment (3.7.1). It seems that between 3.7.1 and 3.9.1 there were considerable changes that caused this. The solution was to use the original “pmm.conf” file and manually adjust if with our internal settings, such as SSL.

Thanks/