Pmm-server upgrade from 2.33 to version 3

1

Hi,

I tried to upgrade pmm-server from version 2.33 to version 3 , but it failed.

docker ps -a shows this

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
71d53c26945e percona/pmm-server:3 “/opt/entrypoint.sh” 2 minutes ago Up 2 minutes (unhealthy) 8080/tcp, 0.0.0.0:443->443/tcp, 8443/tcp pmm-server
1a182730390c percona/pmm-server:2 “/bin/true” 2 years ago Created pmm-data

I followed this link

any help would be appreciated.

2

Adding the logs.

2025-09-22 08:22:17,714 INFO exited: qan-api2 (exit status 1; not expected)
2025-09-22 08:22:18,571 INFO exited: grafana (exit status 1; not expected)
2025-09-22 08:22:18,831 INFO success: pmm-init entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-09-22 08:22:18,831 INFO success: postgresql entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-09-22 08:22:18,831 INFO success: clickhouse entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-09-22 08:22:18,831 INFO success: nginx entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-09-22 08:22:18,831 INFO success: victoriametrics entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-09-22 08:22:18,831 INFO success: vmalert entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-09-22 08:22:18,831 INFO success: vmproxy entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-09-22 08:22:18,832 INFO spawned: ‘qan-api2’ with pid 396
2025-09-22 08:22:18,832 INFO success: pmm-managed entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-09-22 08:22:19,033 INFO exited: qan-api2 (exit status 1; not expected)
2025-09-22 08:22:19,619 INFO spawned: ‘grafana’ with pid 431
2025-09-22 08:22:20,080 INFO exited: grafana (exit status 1; not expected)
2025-09-22 08:22:21,094 INFO spawned: ‘qan-api2’ with pid 536
2025-09-22 08:22:21,094 INFO exited: pmm-init (exit status 0; expected)
2025-09-22 08:22:22,539 INFO spawned: ‘grafana’ with pid 545
2025-09-22 08:22:22,540 INFO success: qan-api2 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-09-22 08:22:23,196 INFO exited: grafana (exit status 1; not expected)
2025-09-22 08:22:26,538 INFO spawned: ‘grafana’ with pid 566
2025-09-22 08:22:27,248 INFO exited: grafana (exit status 1; not expected)
2025-09-22 08:22:31,538 INFO spawned: ‘grafana’ with pid 582
2025-09-22 08:22:31,955 INFO exited: grafana (exit status 1; not expected)
2025-09-22 08:22:37,251 INFO spawned: ‘grafana’ with pid 602
2025-09-22 08:22:37,674 INFO exited: grafana (exit status 1; not expected)
2025-09-22 08:22:44,131 INFO spawned: ‘grafana’ with pid 630
2025-09-22 08:22:45,133 INFO success: grafana entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-09-22 08:22:45,133 INFO exited: grafana (exit status 1; not expected)
2025-09-22 08:22:46,136 INFO spawned: ‘grafana’ with pid 639
2025-09-22 08:22:46,526 INFO exited: grafana (exit status 1; not expected)
2025-09-22 08:22:48,165 INFO spawned: ‘grafana’ with pid 655
2025-09-22 08:22:49,115 INFO exited: grafana (exit status 1; not expected)
2025-09-22 08:22:49,255 INFO spawned: ‘pmm-agent’ with pid 672
2025-09-22 08:22:50,266 INFO success: pmm-agent entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-09-22 08:22:51,436 INFO spawned: ‘grafana’ with pid 716
2025-09-22 08:22:51,884 INFO exited: grafana (exit status 1; not expected)
2025-09-22 08:22:55,728 INFO spawned: ‘grafana’ with pid 791
2025-09-22 08:22:56,359 INFO exited: grafana (exit status 1; not expected)

3

These appear to be only the docker logs. Can you help share logs from PMM server/agents? You might have to copy the logs from the container first

docker cp pmm-server:/srv/logs/ .

Then see if you can share grafana.log, qan-api2.log, pmm-agent.log, and pmm-managed.log

4

grafana.log (2.0 MB)

qan-api2.log (666.2 KB)

pmm-managed.log (1.8 MB)

pmm-agent-updated.log (9.9 MB)

5

Alright, can you also share clickhouse logs? For agent logs - you can inspect it as well and see if there’s a possible error there.

6

clickhouse-server.log (9.3 MB)

7

The only error I see in grafana.log is “Connecting to DB” dbtype=postgres
Error: ✗ pq: role “grafana” does not exist.

And clickhouse-server.log shows:

DB::throwAtAssertionFailed

Due you see any other errors in rest of the files.

8

@Michael_Okoko - Could you please help to find a solution.

Here is something that may help.

[root@pmm-server logs]# docker exec -it pmm-server supervisorctl status
clickhouse RUNNING pid 21, uptime 21:49:25
grafana FATAL Exited too quickly (process log may have details)
nginx RUNNING pid 23, uptime 21:49:25
nomad-server STOPPED Not started
pmm-agent RUNNING pid 672, uptime 21:48:53
pmm-init EXITED Sep 22 08:22 AM
pmm-managed RUNNING pid 28, uptime 21:49:25
postgresql RUNNING pid 20, uptime 21:49:25
qan-api2 RUNNING pid 536, uptime 21:49:21
victoriametrics RUNNING pid 24, uptime 21:49:25
vmalert RUNNING pid 25, uptime 21:49:25
vmproxy RUNNING pid 26, uptime 21:49:25