Yum update Fails on AWS Marketplace Instance

Percona Monitoring and Management (PMM) PMM 2.x
1

Hi,

I’ve been running PMM on the marketplace AWS instance, Oracle Linux, and I’ve done a simple yum update. The web interface helpfully told me some of PMM’s files hadn’t updated or something so I rebooted the instance. It then told me PMM was upgrading and all I get now is an Nginx bad gateway. I’m now at a total loss and this is the second time I’ve had this happen to me. Does no one test this stuff?

Thanks,

2

Grafana seemed to be failing where it couldn’t create a sqlite database. This was in /usr/share/grafana/data/ and was owned by root, so I chowned this to grafana:grafana and this seemed to solve that problem. Then it failed with an error of ‘Failed to start grafana. error: app provisioning error: plugin not installed: “pmm-app”’.

pmm=managed is failing with ‘time=“2024-06-24T12:54:30.725+00:00” level=warning msg=“Get "http://127.0.0.1:3000/api/auth/key\”: dial tcp 127.0.0.1:3000: connect: connection refused" component=grafana/auth prefix=/victoriametrics req=“GET /victoriametrics/api/v1/write”’ presumably because Grafana won’t start.

I’m completely at a loss.

3

Hello @davidl, you performed a few actions that are not recommended or are unnecessary. Our AMI’s are not intended to be accessed/managed via SSH, so running yum update wasn’t something to do. I believe this may have updated specific packages/libraries that are causing conflict with PMM.

We publish and rigorously test our AMIs before releasing to the public. These are tested with specific versions of specific packages that we know works perfectly. You should upgrade the entire AMI when we release new versions of PMM rather than running yum updates.

4

If you want to do a more traditional server setup where ‘yum update’ is a typical thing to do, I suggest you launch your own EC2 instance and run PMM via docker or podman. Since everything is contained in the container image, running yum update on the outer OS will not affect the running container instance.

5

I’m not really all that interested in upgrading the entire AMI. That will mean migrating data, configs etc. or starting from scratch. If you can’t do a simple yum update, apt upgrade or similar this means the setup is in a pretty poor and flakey state. I’m also not that interested in having to run docker either because that presents the same problems. All that should be required is a set of repos just like other software.

6

If you mount an EBS volume, then all data a configs persist through upgrade.

You are entitled to that opinion, however, based on our surveys of PMM users over the past 5+ years, users enjoy the ease-of-use of PMMs AMIs/containers because they don’t have to mess with yum update or other typical OS maintenance tasks.

I disagree. If I’m running PMM inside a container, and in the OS I run ‘yum update’, that will certainly not affect the running container in any way.

“just like other software”, we have 1000s of installations of PMM, which means 1000s of differences: different OS, different versions, etc. It would be near impossible for us to support/manage all of that. For simplicity sake, both ours and our customers, we provide easy-to-use, simple ways to run our software: ready-to-use AMIs, docker images, and VM images.