Percona Monitoring and Management 2.19.0 Released

Percona Monitoring and Management (PMM) PMM 2.x
1

June 30, 2021 - Find release notes at PMM 2.19.0 - Percona Monitoring and Management

2

Stuck in an upgrade loop. What can be done?

ProjectName: pmm-update
Version: 2.18.0
PMMVersion: 2.18.0
Timestamp: 2021-06-01 04:02:49 (UTC)
FullCommit: d3c44721872deb5b1ed143324bb9713943810959
Starting "yum --verbose info installed pmm-update" ...
Loading "changelog" plugin
Loading "fastestmirror" plugin
Loading "ovl" plugin
Config time: 0.010
rpmdb time: 0.000
ovl: Copying up (0) files from OverlayFS lower layer
Yum version: 3.4.3
Installed Packages
Name        : pmm-update
Arch        : noarch
Version     : 2.18.0
Release     : 57.2106010402.d3c4472.el7
Size        : 2.3 M
Repo        : installed
From repo   : pmm2-server
Committer   : Nurlan Moldomurov <nurlan.moldomurov@percona.com>
Committime  : Tue Jun  1 12:00:00 2021
Buildtime   : Tue Jun  1 04:02:50 2021
Install time: Mon Jun 14 05:20:17 2021
Installed by: System <unset>
Changed by  : System <unset>
Summary     : Tool for updating packages and OS configuration for PMM Server
URL         : https://github.com/percona/pmm-update
License     : AGPLv3
Description : Tool for updating packages and OS configuration for PMM Server

Starting "yum update --assumeyes pmm-update" ...
Loaded plugins: changelog, fastestmirror, ovl
Loading mirror speeds from cached hostfile
 * base: mirror.softaculous.com
 * epel: mirror.de.leaseweb.net
 * extras: mirror.softaculous.com
 * updates: mirror.checkdomain.de
Resolving Dependencies
--> Running transaction check
---> Package pmm-update.noarch 0:2.18.0-57.2106010402.d3c4472.el7 will be updated
---> Package pmm-update.noarch 0:2.19.0-59.2106301054.58b0b89.el7 will be an update
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================
 Package      Arch     Version                              Repository     Size
================================================================================
Updating:
 pmm-update   noarch   2.19.0-59.2106301054.58b0b89.el7     pmm2-server   831 k

Transaction Summary
================================================================================
Upgrade  1 Package

Total download size: 831 k
Downloading packages:
Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Updating   : pmm-update-2.19.0-59.2106301054.58b0b89.el7.noarch           1/2 
  Cleanup    : pmm-update-2.18.0-57.2106010402.d3c4472.el7.noarch           2/2 
  Verifying  : pmm-update-2.19.0-59.2106301054.58b0b89.el7.noarch           1/2 
  Verifying  : pmm-update-2.18.0-57.2106010402.d3c4472.el7.noarch           2/2 

Updated:
  pmm-update.noarch 0:2.19.0-59.2106301054.58b0b89.el7                          

Complete!
Starting "yum --verbose info installed pmm-update" ...
Loading "changelog" plugin
Loading "fastestmirror" plugin
Loading "ovl" plugin
Config time: 0.011
rpmdb time: 0.000
ovl: Copying up (0) files from OverlayFS lower layer
Yum version: 3.4.3
Installed Packages
Name        : pmm-update
Arch        : noarch
Version     : 2.19.0
Release     : 59.2106301054.58b0b89.el7
Size        : 2.3 M
Repo        : installed
From repo   : pmm2-server
Committer   : Denys Kondratenko <denys.kondratenko@percona.com>
Committime  : Wed Jun 30 12:00:00 2021
Buildtime   : Wed Jun 30 10:54:26 2021
Install time: Mon Jul  5 10:23:07 2021
Installed by: System <unset>
Changed by  : System <unset>
Summary     : Tool for updating packages and OS configuration for PMM Server
URL         : https://github.com/percona/pmm-update
License     : AGPLv3
Description : Tool for updating packages and OS configuration for PMM Server

time="2021-07-05T10:23:07Z" level=info msg="pmm-update:\nbefore update = {Version:2.18.0 FullVersion:2.18.0-57.2106010402.d3c4472.el7 BuildTime:2021-06-01 04:02:50 +0000 UTC Repo:pmm2-server}\n after update = {Version:2.19.0 FullVersion:2.19.0-59.2106301054.58b0b89.el7 BuildTime:2021-06-30 10:54:26 +0000 UTC Repo:pmm2-server}"
time="2021-07-05T10:23:07Z" level=info msg="Version changed, exiting."

Kind Regards!

3

Alright scratch that. The problem cleared up after a few hours.
It looks like the http package source was offline.

4

Good to hear that the problem is solved!

5

After updating to 2.19.0 all pmm_agents log next text:

Jul 5 08:29:36 qapres02 pmm-agent[79163]: #033[36mINFO#033[0m[2021-07-05T08:29:36.010-07:00] Connecting to https://admin:***@pmm-server.qaserver.com:443/ … #033[36mcomponent#033[0m=client
Jul 5 08:29:36 qapres02 pmm-agent[79163]: #033[36mINFO#033[0m[2021-07-05T08:29:36.015-07:00] Connected to pmm-server.qaserver.com:443. #033[36mcomponent#033[0m=client
Jul 5 08:29:36 qapres02 pmm-agent[79163]: #033[36mINFO#033[0m[2021-07-05T08:29:36.015-07:00] Establishing two-way communication channel … #033[36mcomponent#033[0m=client
Jul 5 08:29:41 qapres02 pmm-agent[79163]: #033[31mERRO#033[0m[2021-07-05T08:29:41.011-07:00] Failed to establish two-way communication channel: context canceled. #033[31mcomponent#033[0m=client
Jul 5 08:29:53 qapres02 pmm-agent[79163]: #033[36mINFO#033[0m[2021-07-05T08:29:53.490-07:00] Connecting to https://admin:***@pmm-server.qaserver.com:443/ … #033[36mcomponent#033[0m=client
Jul 5 08:29:53 qapres02 pmm-agent[79163]: #033[36mINFO#033[0m[2021-07-05T08:29:53.494-07:00] Connected to pmm-server.qaserver.com:443. #033[36mcomponent#033[0m=client
Jul 5 08:29:53 qapres02 pmm-agent[79163]: #033[36mINFO#033[0m[2021-07-05T08:29:53.494-07:00] Establishing two-way communication channel … #033[36mcomponent#033[0m=client
Jul 5 08:29:58 qapres02 pmm-agent[79163]: #033[31mERRO#033[0m[2021-07-05T08:29:58.490-07:00] Failed to establish two-way communication channel: context canceled. #033[31mcomponent#033[0m=client

After rolling back to 2.17.0 - all is fine

6

Hi @Iris

You haven’t mentioned if you also upgraded the clients, were they left on 2.17.0?

C

7

clients were not upgraded

8

OK, so…

  • Did you try upgrading them before reverting the server?
  • Had any customisations been made?
  • Once the server was upgraded did you validate that it was working as expected from the UI? You haven’t mentioned how all of this is deployed (AMI, containers, k8s operator, custom, etc), but supervisorctl status for the server should show if there was a problem with the server itself.
  • Did you export logs from the server (https://<pmm-server>:<port>/logs.zip) to see if there were any error messages that aligned with the requests?

There is information about troubleshooting in the documentation, hopefully that will help you.

If you can reproduce the issue then please create a ticket in Jira