Hi,
I’ve run into a number of issues after updating to PMM 2.27 on an AWS instance. A small selection, together with more info on the instance setup, follows.
Due to a reduction in the number of monitored servers, I shutdown our existing instance, resized it and added an AWS EIP to give it a fixed ip address. On restart, I updated the pmm software via yum, and also updated the client software on 3 centos 7 servers.
Since then, I’ve had to add the entry “proxy_set_header Host $host;” to pmm.conf in nginx, adjusted the error and access.log settings to real file entries rather than /dev/stdout etc, and allowed access to port 7772 so that the web app starts to work. DB analytics, which were working fine, now just say Panel plugin not found: pmm-qan-app-panel, and finally, that old chestnut, 2 removed nodes still appear on the dashboard.
Any suggestions on how to get analytics working again, and if there will be an update or advisory as how to fix things properly. This is not a docker instance, but an AWS box created from the marketplace initially.
Thanks,
Mike
1 Like
Well the nodes you removed will stay on the dashboard until the data about those nodes ages out per your retention policy (30 days by default).
Is there a reason you didn’t run the update via the UI’s upgrade button? If you only pulled the packages down via yum you likely did not run any of the ansible scripts that perform the data migrations, schema updates, etc. Can you confirm what commands you ran to do the upgrade and we’ll get someone internally to take a look and see if there’s a path forward.
1 Like
Sometimes its hard not to just go yum update.
In this particular case, for many reasons, its just going to get blown away.
Having assigned an EIP, the clients are all properly defined, and just have to apply some security groups.
I suspect that disabling the repo as a final part of the upgrade process on RH/Centos etc, might not be a bad idea.
Thanks
1 Like
All done. As the instance has an EIP assigned, and the config files already exist, each node is back up and collecting with just 3 lines.
which are - for Centos 7 at least
pmm-agent setup --config-file=/usr/local/percona/pmm2/config/pmm-agent.yaml --server-address=a.b.c.d --server-insecure-tls --server-username=admin --server-password=xxxx
systemctl start pmm-agent.service
pmm-admin add mysql --username=pmm --password=yyyy
yes, I lost the historical stuff, but at least its a quick recovery.
1 Like
Good to know! I always recommend using a hostname over IP address simply because you don’t have to reregister every single client if your IP ever changes (even if you don’t have DNS readily available, fake it with /etc/hosts entries).
I did talk with one of the DevOps team members and they said in the future you need only run:
/usr/sbin/pmm-update -perform -playbook=/usr/share/pmm-update/ansible/playbook/tasks/update.yml which will run all the missing steps that yum update doesn’t know about.
1 Like