After upgrade from pmm 2.41 to pmm 2.44

Percona Monitoring and Management (PMM)
1

Description:

Problems after upgrade from pmm 2.41.0 to 2.44

Steps to Reproduce:

After the upgrade i’m unable to access to the server both with admin user and personal user

Logs:

supervisorctl status shows

alertmanager RUNNING pid 37, uptime 0:05:10
clickhouse RUNNING pid 19, uptime 0:05:10
dbaas-controller STOPPED Not started
grafana RUNNING pid 29, uptime 0:05:10
nginx RUNNING pid 31, uptime 0:05:10
pmm-agent RUNNING pid 867, uptime 0:04:39
pmm-managed RUNNING pid 50, uptime 0:05:10
pmm-update-perform STOPPED Not started
pmm-update-perform-init EXITED Jan 05 10:37 AM
postgresql RUNNING pid 18, uptime 0:05:10
prometheus STOPPED Not started
qan-api2 RUNNING pid 443, uptime 0:05:09
victoriametrics RUNNING pid 32, uptime 0:05:10
vmalert RUNNING pid 33, uptime 0:05:10
vmproxy RUNNING pid 44, uptime 0:05:10Expected Result:

Additional Information:

Tried to start also with these to reset the password

docker stop pmm-server
docker rm pmm-server
docker run --detach --restart always
–publish 443:443
-v pmm-data:/srv
-e GF_SECURITY_ADMIN_PASSWORD=“NEWPASSWORD”
–name pmm-server percona/pmm-server:2

docker exec -it pmm-server supervisorctl tail pmm-update-perform-init
shows:
PLAY RECAP *********************************************************************
localhost : ok=19 changed=2 unreachable=0 failed=0 skipped=22 rescued=0 ignored=0

2

Hi!

Did you upgrade through the GUI or through a shell?

Do you get invalid password or the login screen does not even shows up? If the latter it might be a problem with PMM itself, not the password.

Regards

3

Hi,
I upgraded stopping the old container and starting the new one on the shared volume docker.
I get invalid password, and the login screen shows up.

4

Without taking a look at the logs is difficult to know what happened.

For example if the old pmm-server was using the shared pmm-data, how the upgrade was run and if it finished successfully etc..

By chance, did you take a backup of PMM before doing the upgrade?

I would suggest you can log in with the pmm admin/admin default password.

Also (hackish) you can consider logging into the PMM-server bash and check what there is under /srv.

Regards

5

After having changed the admin password with:
docker exec -t pmm-server change-admin-password <new_password>

I’ve been able to access and have seen that all history disappeared (so the upgrade has not migrated the old data)
Since i don’t know if (and where) there’s a backup i’ve added again all clients with
pmm-admin config --server-insecure-tls --server-url=xxx --force
systemctl restart pmm-agent.service
and reconfigured services (some galera and mongodb clusters)

Done this metrics have started to appear again, but the space on the volume has not decreased even though old metrics are not anymore visible,
most of space is taken under /pmm/docker/volumes/pmm-data/_data/clickhouse/store/
Is this space going to be cleaned up automatically or must be cleaned up manually?