we are running pmm 2.21.0 and it shows the software as up to date and last check as current date although there is 2.22.0 available.
May be a silly question but does your PMM server have access to the internet to check? Have you checked PMM → Settings → Advanced Settings → “Check for updates” to see if it’s disabled?
I’m running 2.21.0 as well and I do see the Sept 22 update to 2.22.0 in the upgrade widget so seems localized. If neither of those help we can sneak a peak at logs while you hit the refresh button to see if anything jumps out (inside the container it’d be /srv/logs/pmm-managed.log and you’ll be looking for a call to CheckUpdates).
1 Like
Hi @steve.hoffman
Please find the snippet of log [/srv/logs/pmm-managed.log] when we hit refresh button
INFO[2021-10-13T14:17:22.272+00:00] Starting RPC /server.Server/CheckUpdates … request=482ee6df-2c30-11ec-bd52-0242ac110002
WARN[2021-10-13T14:17:23.587+00:00] RPC /server.Server/CheckUpdates done in 1.315098331s (quite long). request=482ee6df-2c30-11ec-bd52-0242ac110002
Please find attached 
1 Like
Hi @phani_g,
Could you give a log of v1/Updates/Check request?
In Chrome browser:
Inspect → network tab → reload browser page → click to refresh version icon.
It should looks like this:
1 Like
I am not able to find the log mentioned, request you to give me the exact path of v1/Update/Check Request
Please find attached
1 Like
Hi @phani_g, thank you!
Hmm, it’s interesting…
Do you use the Docker container or AMI/OVA image?
If you use Docker container can you run:
docker exec <container_name> yum clean all
where <container_name> container with pmm-server (usually pmm-server).
and rerun the check upgrade process.
If you use OVA/AMI then you need to login to server and run:
yum clean all
And also rerun the upgrade process.
1 Like
pmm_user > docker exec 6329caa8bde3 yum clean all
Loaded plugins: changelog, fastestmirror, ovl
Cleaning repos: base epel extras percona-ppg-11 percona-release-noarch
: percona-release-x86_64 pmm2-server prel-release-noarch updates
Cleaning up list of fastest mirrors
Other repos take up 11 M of disk space (use --verbose for details)
No luck. it still shows the same.
1 Like
I am completely stumped…where on earth is your system getting that answer from???
Actually, where in the world are you located…possible we have a bad cache in our CDN that refuses acknowledge the change and is still serving the wrong answer…
1 Like
Hi @Tanuj_Bolisetty
Could you please check to what ip-address repo.percona.com is resolved?
Also please check that the connection is ok to it.
Do you use any mirrors?
1 Like
Servers are located in atlanta georgia
[mysql@pmmhost ~]$ docker exec 6329caa8bde3 ping repo.percona.com
PING us.r53.percona.com (167.99.233.229) 56(84) bytes of data.
64 bytes from 167.99.233.229 (167.99.233.229): icmp_seq=1 ttl=51 time=22.1 ms
64 bytes from 167.99.233.229 (167.99.233.229): icmp_seq=2 ttl=51 time=20.6 ms
64 bytes from 167.99.233.229 (167.99.233.229): icmp_seq=3 ttl=51 time=21.1 ms
64 bytes from 167.99.233.229 (167.99.233.229): icmp_seq=4 ttl=51 time=20.7 ms
1 Like
I could say that it is not CDN issue. I checked and was able to install all packages from that node.
Could you please try executing sudo yum clean all on your pmm server and recheck?
1 Like
could you please also provide output of the:
yum info pmm-update and yum repoinfo
?
1 Like
[root@6329caa8bde3 opt]# yum info pmm-update
Loaded plugins: changelog, fastestmirror, ovl
Determining fastest mirrors
One of the configured repositories failed (Unknown),
and yum doesn’t have enough cached data to continue. At this point the only
safe thing yum can do is fail. There are a few ways to work “fix” this:
1. Contact the upstream for the repository and get them to fix the problem.
2. Reconfigure the baseurl/etc. for the repository, to point to a working
upstream. This is most often useful if you are using a newer
distribution release than is supported by the repository (and the
packages for the previous distribution release still work).
3. Run the command with the repository temporarily disabled
yum --disablerepo=<repoid> ...
4. Disable the repository permanently, so yum won't use it by default. Yum
will then just ignore the repository until you permanently enable it
again or use --enablerepo for temporary usage:
yum-config-manager --disable <repoid>
or
subscription-manager repos --disable=<repoid>
5. Configure the failing repository to be skipped, if it is unavailable.
Note that yum will try to contact the repo. when it runs most commands,
so will have to try and fail each time (and thus. yum will be be much
slower). If it is a very temporary problem though, this is often a nice
compromise:
yum-config-manager --save --setopt=<repoid>.skip_if_unavailable=true
Cannot retrieve metalink for repository: epel/x86_64. Please verify its path and try again
[root@6329caa8bde3 opt]# yum repoinfo
Loaded plugins: changelog, fastestmirror, ovl
Loading mirror speeds from cached hostfile
One of the configured repositories failed (Unknown),
and yum doesn’t have enough cached data to continue. At this point the only
safe thing yum can do is fail. There are a few ways to work “fix” this:
1. Contact the upstream for the repository and get them to fix the problem.
2. Reconfigure the baseurl/etc. for the repository, to point to a working
upstream. This is most often useful if you are using a newer
distribution release than is supported by the repository (and the
packages for the previous distribution release still work).
3. Run the command with the repository temporarily disabled
yum --disablerepo=<repoid> ...
4. Disable the repository permanently, so yum won't use it by default. Yum
will then just ignore the repository until you permanently enable it
again or use --enablerepo for temporary usage:
yum-config-manager --disable <repoid>
or
subscription-manager repos --disable=<repoid>
5. Configure the failing repository to be skipped, if it is unavailable.
Note that yum will try to contact the repo. when it runs most commands,
so will have to try and fail each time (and thus. yum will be be much
slower). If it is a very temporary problem though, this is often a nice
compromise:
yum-config-manager --save --setopt=<repoid>.skip_if_unavailable=true
Cannot retrieve metalink for repository: epel/x86_64. Please verify its path and try again
1 Like
so couple of questions. What your image is based on? Have you done any modifications? Did you upgrade before?
It looks like yum db is broken or something. Here @nikita.beletskii could help you to restore.
@nikita.beletskii any way we could replay some ansible role to restore repos ? Or do we need to do it by hand?
In general @Tanuj_Bolisetty you need to find out what repo is broken and fix it.
Here are the repos I have and content of some of them:
[root@pmm-deployment-67f897c5c-7wpz8 opt]# ls -la /etc/yum.repos.d/
total 92
drwxr-xr-x 1 root root 4096 Sep 21 10:28 .
drwxr-xr-x 1 root root 4096 Oct 19 07:25 ..
-rw-r--r-- 1 root root 1664 Nov 23 2020 CentOS-Base.repo
-rw-r--r-- 1 root root 1309 Nov 23 2020 CentOS-CR.repo
-rw-r--r-- 1 root root 649 Nov 23 2020 CentOS-Debuginfo.repo
-rw-r--r-- 1 root root 314 Nov 23 2020 CentOS-fasttrack.repo
-rw-r--r-- 1 root root 630 Nov 23 2020 CentOS-Media.repo
-rw-r--r-- 1 root root 1331 Nov 23 2020 CentOS-Sources.repo
-rw-r--r-- 1 root root 8515 Nov 23 2020 CentOS-Vault.repo
-rw-r--r-- 1 root root 616 Nov 23 2020 CentOS-x86_64-kernel.repo
-rw-r--r-- 1 root root 177 Sep 21 10:27 clickhouse.repo
-rw-r--r-- 1 root root 1358 Sep 4 17:37 epel.repo
-rw-r--r-- 1 root root 1457 Sep 4 17:37 epel-testing.repo
-rw-r--r-- 1 root root 98 Sep 21 10:24 local.repo
-rw-r--r-- 1 root root 108 Sep 21 10:26 nginx.repo
-rw-r--r-- 1 root root 780 Sep 21 10:28 percona-original-release.repo
-rw-r--r-- 1 root root 780 Sep 21 10:25 percona-original-testing.repo.bak
-rw-r--r-- 1 root root 207 Sep 21 10:23 percona-ppg-11.repo
-rw-r--r-- 1 root root 301 Sep 21 10:23 percona-prel-release.repo
-rw-r--r-- 1 root root 215 Sep 21 10:24 pmm2-server.repo
[root@pmm-deployment-67f897c5c-7wpz8 opt]#
[root@pmm-deployment-67f897c5c-7wpz8 opt]# cat /etc/yum.repos.d/percona-original-release.repo
#
# This repo is managed by "percona-release" utility, do not edit!
#
[percona-release-x86_64]
name = Percona Original release/x86_64 YUM repository
baseurl = http://repo.percona.com/percona/yum/release/$releasever/RPMS/x86_64
enabled = 1
gpgcheck = 1
gpgkey = file:///etc/pki/rpm-gpg/PERCONA-PACKAGING-KEY
[percona-release-noarch]
name = Percona Original release/noarch YUM repository
baseurl = http://repo.percona.com/percona/yum/release/$releasever/RPMS/noarch
enabled = 1
gpgcheck = 1
gpgkey = file:///etc/pki/rpm-gpg/PERCONA-PACKAGING-KEY
[percona-release-sources]
name = Percona Original release/sources YUM repository
baseurl = http://repo.percona.com/percona/yum/release/$releasever/SRPMS
enabled = 0
gpgcheck = 1
gpgkey = file:///etc/pki/rpm-gpg/PERCONA-PACKAGING-KEY
[root@pmm-deployment-67f897c5c-7wpz8 opt]# cat /etc/yum.repos.d/pmm2-server.repo
[pmm2-server]
baseurl = https://repo.percona.com/pmm2-components/yum/release/7/RPMS/x86_64/
enabled = 1
gpgcheck = 1
gpgkey = file:///etc/pki/rpm-gpg/PERCONA-PACKAGING-KEY
name = PMM Server YUM repository - x86_64
1 Like
Interesting issue…It seems that you have a problem with epel repo:
Cannot retrieve metalink for repository: epel/x86_64. Please verify its path and try again
Can you give output from this command, pls:
# cat /etc/yum.repos.d/epel.repo
And these commands too:
yum clean all && yum --disablerepo=epel info ca-certificates
1 Like
[epel]
name=Extra Packages for Enterprise Linux 7 - $basearch
#baseurl=http://download.fedoraproject.org/pub/epel/7/$basearch
metalink=https://mirrors.fedoraproject.org/metalink?repo=epel-7&arch=$basearch&infra=$infra&content=$contentdir
failovermethod=priority
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7
[epel-debuginfo]
name=Extra Packages for Enterprise Linux 7 - $basearch - Debug
#baseurl=http://download.fedoraproject.org/pub/epel/7/$basearch/debug
metalink=https://mirrors.fedoraproject.org/metalink?repo=epel-debug-7&arch=$basearch&infra=$infra&content=$contentdir
failovermethod=priority
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7
gpgcheck=1
[epel-source]
name=Extra Packages for Enterprise Linux 7 - $basearch - Source
#baseurl=mirror.sfo12.us.leaseweb.net | powered by Leaseweb
metalink=https://mirrors.fedoraproject.org/metalink?repo=epel-source-7&arch=$basearch&infra=$infra&content=$contentdir
failovermethod=priority
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7
gpgcheck=1
[root@6329caa8bde3 opt]# yum clean all
Loaded plugins: changelog, fastestmirror, ovl
Cleaning repos: base epel extras percona-ppg-11 percona-release-noarch percona-release-x86_64 pmm2-server prel-release-noarch
: updates
Cleaning up list of fastest mirrors
Other repos take up 11 M of disk space (use --verbose for details)
[root@6329caa8bde3 opt]# yum --disablerepo=epel info ca-certificates
Loaded plugins: changelog, fastestmirror, ovl
Determining fastest mirrors
- base: centos.mirror.constant.com
- extras: mirror.steadfastnet.com
- updates: mirror.siena.edu
base | 3.6 kB 00:00:00
extras | 2.9 kB 00:00:00
percona-ppg-11 | 1.5 kB 00:00:00
percona-release-noarch | 1.5 kB 00:00:00
percona-release-x86_64 | 2.9 kB 00:00:00
https://repo.percona.com/pmm2-components/yum/release/7/RPMS/x86_64/repodata/repomd.xml: [Errno 14] curl#60 - “Peer’s Certificate issuer is not recognized.”
1 Like
Hi @Tanuj_Bolisetty
As I can see you get curl#60 - “Peer’s Certificate issuer is not recognized.”
If you get curl error “(60) peer’s certificate issuer is not recognized” while trying to curl something from another server, it means that your server does not have target’s server trusted certificates. To solve this in a quick way, we can bypass the error simply adding “-k” or “–insecure” to curl
Additionally I would suggest updating ca-certificates package
1 Like
Hm, it looks like you have old root certificates but I can’t reproduce the issue even with the oldest PMM 2 version (PMM 2.0.0).
Do you use any proxies for your PMM server? Which version of PMM docker image are you using?
Can you run this command:
openssl s_client -connect repo.percona.com:443
Also, as Evgeniu said you can try to update ca-certificates package but I didn’t fully understand you have a problem only with our repos or with all repos. Let’s try this command:
yum install --disablerepo percona-ppg-11 --disablerepo epel --disablerepo pmm2-server --disablerepo percona-release-x86_64 --disablerepo percona-release-noarch ca-certificates
It’ll disable our repos and update ca-certificates package.
2 Likes
we are running 2.21.0 image. until now we are able to upgrade fine. we don’t use proxies that I know of.
[root@6329caa8bde3 opt]# openssl s_client -connect repo.percona.com:443
CONNECTED(00000003)
depth=2 C = US, ST = California, O = Zscaler Inc., OU = Zscaler Inc., CN = Zscaler Intermediate Root CA (zscalertwo.net), emailAddress = support@zscaler.com
verify error:num=20:unable to get local issuer certificate
---
Certificate chain
0 s:/OU=Domain Control Validated/CN=*.percona.com
i:/C=US/ST=California/O=Zscaler Inc./OU=Zscaler Inc./CN=Zscaler Intermediate Root CA (zscalertwo.net) (t)
1 s:/C=US/ST=California/O=Zscaler Inc./OU=Zscaler Inc./CN=Zscaler Intermediate Root CA (zscalertwo.net) (t)
i:/C=US/ST=California/O=Zscaler Inc./OU=Zscaler Inc./CN=Zscaler Intermediate Root CA (zscalertwo.net)/emailAddress=support@zscaler.com
2 s:/C=US/ST=California/O=Zscaler Inc./OU=Zscaler Inc./CN=Zscaler Intermediate Root CA (zscalertwo.net)/emailAddress=support@zscaler.com
i:/C=US/ST=California/L=San Jose/O=Zscaler Inc./OU=Zscaler Inc./CN=Zscaler Root CA/emailAddress=support@zscaler.com
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIFAjCCA+qgAwIBAgIIYW7x/1BpFxkwDQYJKoZIhvcNAQELBQAwgY0xCzAJBgNV
BAYTAlVTMRMwEQYDVQQIEwpDYWxpZm9ybmlhMRUwEwYDVQQKEwxac2NhbGVyIElu
Yy4xFTATBgNVBAsTDFpzY2FsZXIgSW5jLjE7MDkGA1UEAxMyWnNjYWxlciBJbnRl
cm1lZGlhdGUgUm9vdCBDQSAoenNjYWxlcnR3by5uZXQpICh0KSAwHhcNMjExMDE2
MDYxNzQxWhcNMjExMDMwMDYxNzQxWjA7MSEwHwYDVQQLExhEb21haW4gQ29udHJv
bCBWYWxpZGF0ZWQxFjAUBgNVBAMMDSoucGVyY29uYS5jb20wggEiMA0GCSqGSIb3
DQEBAQUAA4IBDwAwggEKAoIBAQC0ck4rXhCepjnCTV3b2pxZzV1qqTfjiGp4n6Jw
FxNQrkZsTh7qXfALB8ccVrVlY6eRstKbI66vpuoEs5GiRG5nnFNHLBytlUCVlgE5
0EFN1iAV6vvBBO/qWoDMKltqvwom39fTLnAvNLbkNBvx7TvuJ2NfKdp1QNiYzqE8
mR+2351vo43vpt4btthhLV03IgZ5/5x8sbxhJyyoFsubNzny39cTZubPWHWxZPbS
Qb6a+FZsQNIyPsp7IOorUWMeWjGq6WnBrEaTNBvEljr+2KH+TxKj1phff7P3VVNF
086hMglX5HstUWi8pwkbZh84jZq/3huriQE8EitsNjq3D21LAgMBAAGjggG1MIIB
sTAMBgNVHRMBAf8EAjAAMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjAO
BgNVHQ8BAf8EBAMCBaAwJQYDVR0RBB4wHIINKi5wZXJjb25hLmNvbYILcGVyY29u
YS5jb20wggEEBgorBgEEAdZ5AgQCBIH1BIHyAPAAdgApeb7wnjk5IfBWc59jpXfl
vld9nGAK+PlNXSZcJV3HhAAAAXeBW0pzAAAEAwBHMEUCIBh6UK3XRiFIAyYzYQ5V
WcLTBbL9fbhw3wE6ixtWZH3aAiEAuDriNOAikqiqloT8VTvBOxMW+f5596hXEYVw
eDXmz+UAdgAiRUUHWVUkVpY/oS/x922G4CMmY63AS39dxoNcbuIPAgAAAXeBW0u/
AAAEAwBHMEUCIGx91RcfaysjmRyW3/hls/YwIkBoltfomX2NYqCvJJkKAiEArHZH
KXTXZBQxKzb8lhq+ta9hkkwxq//q1jEfFlTFhmUwQwYDVR0fBDwwOjA4oDagNIYy
aHR0cDovL2dhdGV3YXkuenNjYWxlcnR3by5uZXQvenNjYWxlci16c2NybC0tNC5j
cmwwDQYJKoZIhvcNAQELBQADggEBAB7U7TWg680Qn77rmVYnDjpdPDSLJ/n0+avk
WNEdQ3YyMeniHpSNxQmrFetgOCsYv7NMGUorwBUPESIN2U0fvzK3JwqJJVwpz93A
7XXGV9nDKLCqPAPlgaxY3+TglrzKW4u8HKU1U85eoMdUP5CHlTn/c2liap3F51oW
xKWQBHbjrZ/65y0N57WIxhKEfYbNsG2R08Om1kZM59rDqEaJAEHX+CMVPluItukM
gz1KZq6HDPK6ZVWE1bTizISfvSL8u4IXuVt27mt6ES/0jW2LyMpk7oQfICPK6lEp
8rpYY7ooblBOdXY9WY1ZqKcC+Br1QtK3usH/LsAHaPOrtkbsFQ4=
-----END CERTIFICATE-----
subject=/OU=Domain Control Validated/CN=*.percona.com
issuer=/C=US/ST=California/O=Zscaler Inc./OU=Zscaler Inc./CN=Zscaler Intermediate Root CA (zscalertwo.net) (t)
---
No client certificate CA names sent
Peer signing digest: SHA512
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 3995 bytes and written 415 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES256-GCM-SHA384
Session-ID: 6CD0F9246B33DC3E2F0F727538836523F9B06A8E330EC9673FE48DE8FE15E501
Session-ID-ctx:
Master-Key: 38C18C984D9991B302349CFB20965A037C46620EA1E8013583EB53735156A79FA663B23D90D278A2A2FBC4F42785DA76
Key-Arg : None
Krb5 Principal: None
PSK identity: None
PSK identity hint: None
Start Time: 1634667594
Timeout : 300 (sec)
Verify return code: 20 (unable to get local issuer certificate)
---
Do u want me to run this inside docker ?
yum install --disablerepo percona-ppg-11 --disablerepo epel --disablerepo pmm2-server --disablerepo percona-release-x86_64 --disablerepo percona-release-noarch ca-certificates
1 Like
Yes, all commands that I ask to run must be run inside the container 
1 Like