Description:
When I try to upgrade PMM from 2.39.0 to 2.42.0, my network is broken, and when the network comes back, the upgrade log reminds me error happened. The last line of log is saying something like “start yum-update xxx”. And when I try to reload the page, it just display the following message:
If you're seeing this Grafana has failed to load its application files
1. This could be caused by your reverse proxy settings.
2. If you host grafana under subpath make sure your grafana.ini root_url setting includes subpath. If not using a reverse proxy make sure to set serve_from_sub_path to true.
3. If you have a local dev build make sure you build frontend using: yarn start, yarn start:hot, or yarn build
4. Sometimes restarting grafana-server can help
5. Check if you are using a non-supported browser. For more information, refer to the list of supported browsers.
Steps to Reproduce:
possible way to reproduce:
- use the PMM UI to do the upgrade
- Disable the network
Version:
2.39.0
Logs:
[If applicable, include any relevant log files or error messages]
Expected Result:
Someone tell me how to proceed to finish the upgrade to rollback.
Actual Result:
[What actually happened when the user encountered the issue]
Additional Information:
[Include any additional information that could be helpful to diagnose the issue, such as browser or device information]
UPDATE:
I think the root cause for the update failure is because the repo installed in the docker does not exist.
[root@e9249b32b508 opt]# find /etc/yum.repos.d/* |xargs grep -in 'mirrorlist.centos.org'
/etc/yum.repos.d/CentOS-Base.repo:15:mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=os&infra=$infra
/etc/yum.repos.d/CentOS-Base.repo:23:mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=updates&infra=$infra
/etc/yum.repos.d/CentOS-Base.repo:31:mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=extras&infra=$infra
/etc/yum.repos.d/CentOS-Base.repo:39:mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=centosplus&infra=$infra
/etc/yum.repos.d/CentOS-fasttrack.repo:5:mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=fasttrack&infra=$infra
/etc/yum.repos.d/CentOS-x86_64-kernel.repo:3:mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=kernel&infra=$infra
/etc/yum.repos.d/CentOS-x86_64-kernel.repo:11:mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=experimental&infra=$infra
The mirrorlist.centos.org does not exist anymore.
Starting "yum update --assumeyes pmm-update" ...
Loaded plugins: changelog, fastestmirror, ovl
Loading mirror speeds from cached hostfile
Could not retrieve mirrorlist http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=os&infra=container error was
14: curl#6 - "Could not resolve host: mirrorlist.centos.org; Unknown error"
[root@e9249b32b508 opt]# ping mirrorlist.centos.org
ping: mirrorlist.centos.org: Name or service not known
And I change it mirror.centos.org, but no luck:
[root@e9249b32b508 opt]# ping mirror.centos.org
PING mirror.centos.org (104.206.99.122) 56(84) bytes of data.
64 bytes from 104.206.99.122 (104.206.99.122): icmp_seq=1 ttl=49 time=33.7 ms
64 bytes from 104.206.99.122 (104.206.99.122): icmp_seq=2 ttl=49 time=33.6 ms
64 bytes from 104.206.99.122 (104.206.99.122): icmp_seq=3 ttl=49 time=33.3 ms
Still fail:
failure: repodata/repomd.xml from base: [Errno 256] No more mirrors to try.
http://mirror.centos.org/centos/7/os/x86_64/repodata/repomd.xml: [Errno 14] HTTP Error 404 - Not Found
time="2024-08-21T08:09:32Z" level=fatal msg="Update failed: `yum update --assumeyes pmm-update` failed: exit status 1"
ProjectName: pmm-update
UPDATE:
After remove the broken repo file, upgrade finally succeeded. But the initial upgrade does break the grafana. I have to copy the grafana configuration file from production host to the target docker, and can the grafana be started.
Centos7 repos were moved to vault once Centos7 reached EOL dates.
So in order to fix any issues connected with this it is needed to rewrite baseurl to point to vault
Something like this
sed -i 's/mirrorlist/#mirrorlist/g' /etc/yum.repos.d/CentOS-*
sed -i 's|#\s*baseurl=http://mirror.centos.org|baseurl=http://vault.centos.org|g' /etc/yum.
repos.d/CentOS-*