After upgrading from 2.41.1 to 2.41.2 PostgreSQL DB(Grafana DB) is empty inside PMM server container

[Naresh9999]

Hi Team,

After, upgrading from the 2.41.1 to 2.41.2 version, the postgresql grafana db is dropped and re created during the upgrade and we are unable to login with the admin user and we are getting the below error. And the agents are failing to connect to the PMM server.

From the client server: pmm-admin list

logger=context t=2024-05-09T18:42:08.76901763Z level=error msg=“invalid API key” error=“invalid API key” traceID=
logger=context t=2024-05-09T18:42:08.769048192Z level=error msg=“invalid API key” error=“invalid API key” traceID=
logger=context userId=0 orgId=0 uname= t=2024-05-09T18:42:08.769395906Z level=info msg=“Request Completed” method=GET path=/api/auth/key status=401 remote_addr=127.0.0.1 time_ms=0 duration=733.963µs size=67 referer=
logger=context userId=0 orgId=0 uname= t=2024-05-09T18:42:08.769429508Z level=info msg=“Request Completed” method=GET path=/api/auth/key status=401 remote_addr=127.0.0.1 time_ms=0 duration=878.684µs size=67 referer=
logger=context t=2024-05-09T18:42:08.770758403Z level=error msg=“invalid API key” error=“invalid API key” traceID=
logger=context t=2024-05-09T18:42:08.770861196Z level=error msg=“invalid API key” error=“invalid API key” traceID=
logger=context userId=0 orgId=0 uname= t=2024-05-09T18:42:08.771147756Z level=info msg=“Request Completed” method=GET path=/api/auth/key status=401 remote_addr=127.0.0.1 time_ms=0 duration=661.109µs size=67 referer=
logger=context userId=0 orgId=0 uname= t=2024-05-09T18:42:08.771211609Z level=info msg=“Request Completed” method=GET path=/api/auth/key status=401 remote_addr=127.0.0.1 time_ms=0 duration=726.967µs size=67 referer=
logger=context t=2024-05-09T18:42:08.958594199Z level=error msg=“invalid API key” error=“invalid API key” traceID=
logger=context userId=0 orgId=0 uname= t=2024-05-09T18:42:08.95890645Z level=info msg=“Request Completed” method=GET path=/api/auth/key status=401 remote_addr=127.0.0.1 time_ms=0 duration=901.078µs size=67 referer=
logger=context t=2024-05-09T18:42:08.95996779Z level=error msg=“invalid API key” error=“invalid API key” traceID=
logger=context userId=0 orgId=0 uname= t=2024-05-09T18:42:08.960249371Z level=info msg=“Request Completed” method=GET path=/api/auth/key status=401 remote_addr=127.0.0.1 time_ms=0 duration=594.3µs size=67 referer=
logger=context t=2024-05-09T18:42:09.011087399Z level=error msg=“invalid API key” error=“invalid API key” traceID=
logger=context t=2024-05-09T18:42:09.011283816Z level=error msg=“invalid API key” error=“invalid API key” traceID=
logger=context userId=0 orgId=0 uname= t=2024-05-09T18:42:09.011487551Z level=info msg=“Request Completed” method=GET path=/api/auth/key status=401 remote_addr=127.0.0.1 time_ms=0 duration=837.27µs size=67 referer=
logger=context userId=0 orgId=0 uname= t=2024-05-09T18:42:09.011685679Z level=info msg=“Request Completed” method=GET path=/api/auth/key status=401 remote_addr=127.0.0.1 time_ms=0 duration=901.287µs size=67 referer=
logger=context t=2024-05-09T18:42:09.01281536Z level=error msg=“invalid API key” error=“invalid API key” traceID=
logger=context t=2024-05-09T18:42:09.012840596Z level=error msg=“invalid API key” error=“invalid API key” traceID=
logger=context userId=0 orgId=0 uname= t=2024-05-09T18:42:09.013180265Z level=info msg=“Request Completed” method=GET path=/api/auth/key status=401 remote_addr=127.0.0.1 time_ms=0 duration=739.759µs size=67 referer=
logger=context userId=0 orgId=0 uname= t=2024-05-09T18:42:09.01323109Z level=info msg=“Request Completed” method=GET path=/api/auth/key status=401 remote_addr=127.0.0.1 time_ms=0 duration=902.065µs size=67 referer=
logger=context t=2024-05-09T18:42:09.026801808Z level=error msg=“invalid API key” error=“invalid API key” traceID=
logger=context t=2024-05-09T18:42:09.026913894Z level=error msg=“invalid API key” error=“invalid API key” traceID=
logger=context userId=0 orgId=0 uname= t=2024-05-09T18:42:09.02705431Z level=info msg=“Request Completed” method=GET path=/api/auth/key status=401 remote_addr=127.0.0.1 time_ms=0 duration=606.052µs size=67 referer=
logger=context userId=0 orgId=0 uname= t=2024-05-09T18:42:09.02719217Z level=info msg=“Request Completed” method=GET path=/api/auth/key status=401 remote_addr=127.0.0.1 time_ms=0 duration=673.799µs size=67 referer=
logger=context t=2024-05-09T18:42:09.02790978Z level=error msg=“invalid API key” error=“invalid API key” traceID=
logger=context t=2024-05-09T18:42:09.028121372Z level=error msg=“invalid API key” error=“invalid API key” traceID=
logger=context userId=0 orgId=0 uname= t=2024-05-09T18:42:09.02815091Z level=info msg=“Request Completed” method=GET path=/api/auth/key status=401 remote_addr=127.0.0.1 time_ms=0 duration=520.755µs size=67 referer=
logger=context userId=0 orgId=0 uname= t=2024-05-09T18:42:09.028410929Z level=info msg=“Request Completed” method=GET path=/api/auth/key status=401 remote_addr=127.0.0.1 time_ms=0 duration=571.65µs size=67 referer=
logger=context t=2024-05-09T18:42:09.061106491Z level=error msg=“invalid API key” error=“invalid API key” traceID=
logger=context userId=0 orgId=0 uname= t=2024-05-09T18:42:09.061371115Z level=info msg=“Request Completed” method=GET path=/api/auth/key status=401 remote_addr=127.0.0.1 time_ms=0 duration=676.749µs size=67 referer=
logger=context t=2024-05-09T18:42:09.062231869Z level=error msg=“invalid API key” error=“invalid API key” traceID=
logger=context userId=0 orgId=0 uname= t=2024-05-09T18:42:09.062548858Z level=info msg=“Request Completed” method=GET path=/api/auth/key status=401 remote_addr=127.0.0.1 time_ms=0 duration=560.562µs size=67 referer=
(END)

[Naresh9999]

Can someone please help me?

[Naresh9999]

Inside the PMM container:

[nurlan]

pmm-admin command inside PMM Server will always show that error.
There might be several reasons why it’s happening:

1. Something wrong happened during upgrade, please check logs in /srv/logs/pmm-update-*
2. You tried to log in with wrong credentials multiple times and it lead to user block in Grafana. it happens sometimes.

You can check users table in grafana DB to check if your user is blocked.

[Naresh9999]

@nurlan After upgrading, my entire Postgres database became empty, and I am seeing zero counts in all the tables.

Not sure during the upgrade, why we dropping the Grafana schema as per the upgrade logs.

grafana=# select * from user;
user

postgres
(1 row)
grafana=# select * from api_key;
id | org_id | name | key | role | created | updated | expires | service_account_id | last_used_at | is_revoked
----±-------±-----±----±-----±--------±--------±--------±-------------------±-------------±-----------
(0 rows)

grafana=#

grafana=# select * from user_role;
id | org_id | user_id | role_id | created
----±-------±--------±--------±--------
(0 rows)

grafana=# select * from preferences;
id | org_id | user_id | version | home_dashboard_id | timezone | theme | created | updated | team_id | week_start | json_data
----±-------±--------±--------±------------------±---------±------±--------±--------±--------±-----------±----------
(0 rows)

grafana=# select * from alert_rule;
id | org_id | title | condition | data | updated | interval_seconds | version | uid | namespace_uid | rule_group | no_data_state | exec_err_state | for | annotations | labels | dashboard_u
id | panel_id | rule_group_idx
----±-------±------±----------±-----±--------±-----------------±--------±----±--------------±-----------±--------------±---------------±----±------------±-------±-----------
—±---------±---------------
(0 rows)

[Naresh9999]

Installed by : System
Summary : Tool for updating packages and OS configuration for PMM Server
URL : GitHub - percona/pmm: Percona Monitoring and Management: an open source database monitoring, observability and management tool
License : AGPLv3
Description : Tool for updating packages and OS configuration for PMM Server

Starting “yum update --assumeyes pmm-update” …
Last metadata expiration check: 1 day, 3:21:41 ago on Wed 08 May 2024 03:05:52 PM UTC.
Dependencies resolved.

Package Arch Version Repository Size

Upgrading:
pmm-update noarch 2.41.2-67.2403192109.10aa17a.el9 pmm2-server 921 k

Transaction Summary

Upgrade 1 Package

Total download size: 921 k
Downloading Packages:
pmm-update-2.41.2-67.2403192109.10aa17a.el9.noa 2.0 MB/s | 921 kB 00:00

Total 2.0 MB/s | 921 kB 00:00
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
Preparing : 1/1
Upgrading : pmm-update-2.41.2-67.2403192109.10aa17a.el9.noarch 1/2
Cleanup : pmm-update-2.41.1-67.2401291332.f4004ea.el9.noarch 2/2
Verifying : pmm-update-2.41.2-67.2403192109.10aa17a.el9.noarch 1/2
Verifying : pmm-update-2.41.1-67.2401291332.f4004ea.el9.noarch 2/2

Upgraded:
pmm-update-2.41.2-67.2403192109.10aa17a.el9.noarch

Complete!
Starting “yum --verbose info installed pmm-update” …
Loaded plugins: builddep, changelog, config-manager, copr, debug, debuginfo-install, download, generate_completion_cache, groups-manager, needs-restarting, playground, repoclosure, repodiff, repograph, repomanage, reposync, system-upgrade
YUM version: 4.14.0
cachedir: /var/cache/dnf
Unknown configuration option: async = 1 in /etc/yum.repos.d/local.repo
Unknown configuration option: async = 1 in /etc/yum.repos.d/pmm2-server.repo
User-Agent: constructed: ‘libdnf (Oracle Linux Server 9.3; server; Linux.x86_64)’
Installed Packages
Name : pmm-update
Version : 2.41.2
Release : 67.2403192109.10aa17a.el9
Architecture : noarch
Size : 2.4 M
Source : pmm-update-2.41.2-67.2403192109.10aa17a.el9.src.rpm
Repository : @System
From repo : pmm2-server
Packager : None
Buildtime : Tue 19 Mar 2024 09:09:25 PM UTC
Install time : Thu 09 May 2024 06:27:34 PM UTC
Installed by : System
Summary : Tool for updating packages and OS configuration for PMM Server
URL : GitHub - percona/pmm: Percona Monitoring and Management: an open source database monitoring, observability and management tool
License : AGPLv3
Description : Tool for updating packages and OS configuration for PMM Server

time=“2024-05-09T18:27:35Z” level=info msg=“pmm-update:\nbefore update = {Version:2.41.1 FullVersion:2.41.1-67.2401291332.f4004ea.el9 BuildTime:2024-01-29 13:32:27 +0000 UTC Repo:pmm2-server}\n after update = {Version:2.41.2 FullVersion:2.41.2-67.2403192109.10aa17a.el9 BuildTime:2024-03-19 21:09:25 +0000 UTC Repo:pmm2-server}”
time=“2024-05-09T18:27:35Z” level=info msg=“Version changed, exiting.”
ProjectName: pmm-update
Version: 2.41.2
PMMVersion: 2.41.2
Timestamp: 2024-03-19 21:09:27 (UTC)
FullCommit: 10aa17ad6fe680a99f253cca8b32d3beb04050f6
Starting “yum --verbose info installed pmm-update” …
Loaded plugins: builddep, changelog, config-manager, copr, debug, debuginfo-install, download, generate_completion_cache, groups-manager, needs-restarting, playground, repoclosure, repodiff, repograph, repomanage, reposync, system-upgrade
YUM version: 4.14.0
cachedir: /var/cache/dnf
Unknown configuration option: async = 1 in /etc/yum.repos.d/local.repo
Unknown configuration option: async = 1 in /etc/yum.repos.d/pmm2-server.repo
User-Agent: constructed: ‘libdnf (Oracle Linux Server 9.3; server; Linux.x86_64)’
Installed Packages
Name : pmm-update
Version : 2.41.2
Release : 67.2403192109.10aa17a.el9
Architecture : noarch
Size : 2.4 M
Source : pmm-update-2.41.2-67.2403192109.10aa17a.el9.src.rpm
Repository : @System
From repo : pmm2-server
Packager : None
Buildtime : Tue 19 Mar 2024 09:09:25 PM UTC
Install time : Thu 09 May 2024 06:27:34 PM UTC
Installed by : System
Summary : Tool for updating packages and OS configuration for PMM Server
URL : GitHub - percona/pmm: Percona Monitoring and Management: an open source database monitoring, observability and management tool
License : AGPLv3
Description : Tool for updating packages and OS configuration for PMM Server

Starting “yum update --assumeyes pmm-update” …
Last metadata expiration check: 1 day, 3:21:44 ago on Wed 08 May 2024 03:05:52 PM UTC.
Dependencies resolved.
Nothing to do.
Complete!
Starting “yum --verbose info installed pmm-update” …
Loaded plugins: builddep, changelog, config-manager, copr, debug, debuginfo-install, download, generate_completion_cache, groups-manager, needs-restarting, playground, repoclosure, repodiff, repograph, repomanage, reposync, system-upgrade
YUM version: 4.14.0
cachedir: /var/cache/dnf
Unknown configuration option: async = 1 in /etc/yum.repos.d/local.repo
Unknown configuration option: async = 1 in /etc/yum.repos.d/pmm2-server.repo
User-Agent: constructed: ‘libdnf (Oracle Linux Server 9.3; server; Linux.x86_64)’
Installed Packages
Name : pmm-update
Version : 2.41.2
Release : 67.2403192109.10aa17a.el9
Architecture : noarch
Size : 2.4 M
Source : pmm-update-2.41.2-67.2403192109.10aa17a.el9.src.rpm
Repository : @System
From repo : pmm2-server
Packager : None
Buildtime : Tue 19 Mar 2024 09:09:25 PM UTC
Install time : Thu 09 May 2024 06:27:34 PM UTC
Installed by : System
Summary : Tool for updating packages and OS configuration for PMM Server
URL : GitHub - percona/pmm: Percona Monitoring and Management: an open source database monitoring, observability and management tool
License : AGPLv3
Description : Tool for updating packages and OS configuration for PMM Server

time=“2024-05-09T18:27:37Z” level=info msg=“pmm-update:\nbefore update = {Version:2.41.2 FullVersion:2.41.2-67.2403192109.10aa17a.el9 BuildTime:2024-03-19 21:09:25 +0000 UTC Repo:pmm2-server}\n after update = {Version:2.41.2 FullVersion:2.41.2-67.2403192109.10aa17a.el9 BuildTime:2024-03-19 21:09:25 +0000 UTC Repo:pmm2-server}”
time=“2024-05-09T18:27:37Z” level=info msg=“Version did not change.”
Starting “ansible-playbook --flush-cache /usr/share/pmm-update/ansible/playbook/tasks/update.yml” …
[WARNING]: provided hosts list is empty, only localhost is available. Note that
the implicit localhost does not match ‘all’

PLAY [localhost] ***************************************************************

TASK [Gathering Facts] *********************************************************
ok: [localhost]

TASK [detect /srv/pmm-distribution] ********************************************
ok: [localhost]

TASK [detect containers] *******************************************************
ok: [localhost]

TASK [force container] *********************************************************
skipping: [localhost]

TASK [Enable maintenance mode] *************************************************
changed: [localhost]

TASK [Remove percona-dashboard without architecture] ***************************
ok: [localhost]

TASK [Delete experimental repo file in 2.16 version] ***************************
ok: [localhost]

TASK [Update percona-dashboards package] ***************************************
changed: [localhost]

TASK [Cleanup yum metadata] ****************************************************
skipping: [localhost]

TASK [Cleanup yum metadata] ****************************************************
changed: [localhost]

TASK [Download pmm2 packages] **************************************************
changed: [localhost]

TASK [Update pmm2 packages] ****************************************************
changed: [localhost]

TASK [Create supervisord dir] **************************************************
ok: [localhost]

TASK [Upgrade supervisor config] ***********************************************
changed: [localhost]

TASK [Generate new supervisor config] ******************************************
changed: [localhost]

TASK [Disable pmm-update-perform-init] *****************************************
changed: [localhost]

TASK [Upgrade supervisord config] **********************************************
ok: [localhost]

TASK [Remove supervisord] ******************************************************
skipping: [localhost]

TASK [Stop systemd pmm-agent service, if running] ******************************
skipping: [localhost]

TASK [Create grafana config] ***************************************************

TASK [grafana : Create grafana directory] **************************************
ok: [localhost] => (item=/srv/grafana)
ok: [localhost] => (item=/srv/grafana/plugins)

TASK [grafana : Set Grafana folder for plugins on /srv partition for all users] ***
ok: [localhost]

TASK [grafana : Copy new version of grafana.ini] *******************************
changed: [localhost]

TASK [grafana : Check that the SQLite grafana database exists] *****************
ok: [localhost]

TASK [grafana : Remove database options (SQLite is default)] *******************
changed: [localhost]

TASK [grafana : Remove database host] ******************************************
changed: [localhost]

TASK [grafana : Remove database user] ******************************************
changed: [localhost]

TASK [grafana : Remove database password] **************************************
changed: [localhost]

TASK [grafana : Create provisioning directory] *********************************
ok: [localhost] => (item=datasources)
ok: [localhost] => (item=plugins)
ok: [localhost] => (item=dashboards)

TASK [grafana : Copy grafana provisioning files] *******************************
ok: [localhost] => (item=datasources)
ok: [localhost] => (item=plugins)
ok: [localhost] => (item=dashboards)

TASK [grafana : Upgrade grafana database (Get the latest schema)] **************
changed: [localhost]

TASK [Install postgres] ********************************************************

TASK [postgres : Add PostgreSQL 14 YUM repository for EL7] *********************
skipping: [localhost]

TASK [postgres : Add PostgreSQL 11 YUM repository for EL7] *********************
skipping: [localhost]

TASK [postgres : Install Postgres] *********************************************
skipping: [localhost]

TASK [postgres : Add PostgreSQL 14 YUM repository for EL9] *********************
ok: [localhost]

TASK [postgres : Add PostgreSQL 11 YUM repository for EL9] *********************
ok: [localhost]

TASK [postgres : Install Postgres] *********************************************
ok: [localhost]

TASK [postgres : Create Postgres log file] *************************************
changed: [localhost]

TASK [postgres : Add the user ‘postgres’ to group ‘pmm’] ***********************
ok: [localhost]

TASK [postgres : Check if Postgres 14 directory exists] ************************
ok: [localhost]

TASK [postgres : Create Postgres data dir] *************************************
skipping: [localhost]

TASK [postgres : Initialize Postgres database] *********************************
skipping: [localhost]

TASK [postgres : Check that supervisor socket exists (Upgrade or init)] ********
skipping: [localhost]

TASK [postgres : Stop pmm-managed and postgres before backup database | EL7] ***
skipping: [localhost] => (item=pmm-managed)
skipping: [localhost] => (item=postgresql)
skipping: [localhost]

TASK [postgres : Stop pmm-managed and postgres before backup database | EL9] ***
skipping: [localhost] => (item=pmm-managed)
skipping: [localhost] => (item=postgresql)
skipping: [localhost]

TASK [postgres : Run Postgres database without supervisor] *********************
skipping: [localhost]

TASK [postgres : Dump pmm-managed database] ************************************
skipping: [localhost]

TASK [postgres : Stop Postgres database without supervisor] ********************
skipping: [localhost]

TASK [postgres : Start Postgres 14 database without supervisor] ****************
skipping: [localhost]

TASK [postgres : Create pmm-managed database] **********************************
skipping: [localhost]

TASK [postgres : PostgresDB | Create pmm-managed user] **********
skipping: [localhost]

TASK [postgres : Restore pmm-managed database] *********************************
skipping: [localhost]

TASK [postgres : Check pg_stat_statements extension] ***************************
skipping: [localhost]

TASK [postgres : Stop Postgres 14 database without supervisor] *****************
skipping: [localhost]

TASK [postgres : Rename old Postgres directory] ********************************
skipping: [localhost]

TASK [postgres : Remove old Postgres directory] ********************************
skipping: [localhost]

TASK [postgres : Reread supervisord configuration EL7] *************************
skipping: [localhost]

TASK [postgres : Reread supervisord configuration EL9] *************************
skipping: [localhost]

TASK [postgres : Restart Postgres | EL7] ***************************************
skipping: [localhost] => (item=stop)
skipping: [localhost] => (item=remove)
skipping: [localhost] => (item=add)
skipping: [localhost]

TASK [postgres : Restart Postgres | EL9] ***************************************
skipping: [localhost] => (item=stop)
skipping: [localhost] => (item=remove)
skipping: [localhost] => (item=add)
skipping: [localhost]

TASK [postgres : Run pmm-managed again | EL7] **********************************
skipping: [localhost]

TASK [postgres : Run pmm-managed again | EL9] **********************************
skipping: [localhost]

TASK [Configure systemd] *******************************************************
skipping: [localhost]

TASK [Remove old supervisord service configuration] ****************************
skipping: [localhost]

TASK [Enable supervisord | Make the service persist between reboots] ****
skipping: [localhost]

TASK [Supervisord start | Start supervisord service for AMI/OVF] ********
skipping: [localhost]

TASK [Check that supervisor socket exists] *************************************
ok: [localhost]

TASK [Supervisord start EL7 | Start supervisord for docker] ************
skipping: [localhost]

TASK [Supervisord start EL9 | Start supervisord for docker] *************
skipping: [localhost]

TASK [Wait until postgres port is present before continuing] *******************
ok: [localhost]

TASK [Run initialization playbook] *********************************************

TASK [initialization : Determine type of upgrade] ******************************
skipping: [localhost]

TASK [initialization : Get current version] ************************************
ok: [localhost]

TASK [initialization : Get image version] **************************************
ok: [localhost]

TASK [initialization : Set current version if VERSION doesn’t exist] ***********
skipping: [localhost]

TASK [initialization : Setting current PMM version] ****************************
ok: [localhost]

TASK [initialization : Setting current PMM image version] **********************
ok: [localhost]

TASK [initialization : Print current PMM and image versions] *******************
ok: [localhost] => {
“msg”: “Current version: 2.41.1 Image Version: 2.41.2”
}

TASK [initialization : Determine type of upgrade] ******************************
ok: [localhost]

TASK [initialization : Print Docker upgrade fact] ******************************
ok: [localhost] => {
“msg”: “Docker upgrade: False”
}

TASK [initialization : Enable maintenance mode only for docker upgrade] ********
skipping: [localhost]

TASK [initialization : Update repository settings] *****************************
skipping: [localhost]

TASK [initialization : Clean yum metadata] *************************************
changed: [localhost]

TASK [initialization : Create clickhouse data directory] ***********************
changed: [localhost]

TASK [initialization : Check that conversion flag isn’t created] ***************
ok: [localhost]

TASK [initialization : Create clickhouse data directory] ***********************
ok: [localhost]

TASK [initialization : Create empty file to convert clickhouse databases from ordinary to atomic] ***
changed: [localhost]

TASK [initialization : Restart clickhouse EL7] *********************************
skipping: [localhost]

TASK [initialization : Restart clickhouse EL9] *********************************
changed: [localhost]

TASK [Upgrade dashboards] ******************************************************

TASK [dashboards_upgrade : Check that old plugin dir exists] *******************
ok: [localhost]

TASK [dashboards_upgrade : Find custom plugins in old plugin dir] **************
skipping: [localhost]

TASK [dashboards_upgrade : Synchronization plugins (for versions before 2.22.0)] ***
skipping: [localhost]

TASK [dashboards_upgrade : Find custom plugin in old plugin dir] ***************
skipping: [localhost]

TASK [dashboards_upgrade : Get plugin list] ************************************
ok: [localhost]

TASK [dashboards_upgrade : Delete existing dist folder] ************************
changed: [localhost] => (item={‘path’: ‘/usr/share/percona-dashboards/panels/camptocamp-prometheus-alertmanager-datasource’, ‘mode’: ‘0755’, ‘isdir’: True, ‘ischr’: False, ‘isblk’: False, ‘isreg’: False, ‘isfifo’: False, ‘islnk’: False, ‘issock’: False, ‘uid’: 998, ‘gid’: 998, ‘size’: 146, ‘inode’: 1140854054, ‘dev’: 49, ‘nlink’: 1, ‘atime’: 1709552496.0, ‘mtime’: 1715279282.4608197, ‘ctime’: 1715279282.4608197, ‘gr_name’: ‘grafana’, ‘pw_name’: ‘grafana’, ‘wusr’: True, ‘rusr’: True, ‘xusr’: True, ‘wgrp’: False, ‘rgrp’: True, ‘xgrp’: True, ‘woth’: False, ‘roth’: True, ‘xoth’: True, ‘isuid’: False, ‘isgid’: False})
changed: [localhost] => (item={‘path’: ‘/usr/share/percona-dashboards/panels/grafana-clickhouse-datasource’, ‘mode’: ‘0755’, ‘isdir’: True, ‘ischr’: False, ‘isblk’: False, ‘isreg’: False, ‘isfifo’: False, ‘islnk’: False, ‘issock’: False, ‘uid’: 998, ‘gid’: 998, ‘size’: 4096, ‘inode’: 1275071961, ‘dev’: 49, ‘nlink’: 1, ‘atime’: 1709552496.0, ‘mtime’: 1715279282.4618196, ‘ctime’: 1715279282.4618196, ‘gr_name’: ‘grafana’, ‘pw_name’: ‘grafana’, ‘wusr’: True, ‘rusr’: True, ‘xusr’: True, ‘wgrp’: False, ‘rgrp’: True, ‘xgrp’: True, ‘woth’: False, ‘roth’: True, ‘xoth’: True, ‘isuid’: False, ‘isgid’: False})
changed: [localhost] => (item={‘path’: ‘/usr/share/percona-dashboards/panels/grafana-piechart-panel’, ‘mode’: ‘0755’, ‘isdir’: True, ‘ischr’: False, ‘isblk’: False, ‘isreg’: False, ‘isfifo’: False, ‘islnk’: False, ‘issock’: False, ‘uid’: 998, ‘gid’: 998, ‘size’: 4096, ‘inode’: 1476397821, ‘dev’: 49, ‘nlink’: 1, ‘atime’: 1709552496.0, ‘mtime’: 1715279282.4638195, ‘ctime’: 1715279282.4638195, ‘gr_name’: ‘grafana’, ‘pw_name’: ‘grafana’, ‘wusr’: True, ‘rusr’: True, ‘xusr’: True, ‘wgrp’: False, ‘rgrp’: True, ‘xgrp’: True, ‘woth’: False, ‘roth’: True, ‘xoth’: True, ‘isuid’: False, ‘isgid’: False})
changed: [localhost] => (item={‘path’: ‘/usr/share/percona-dashboards/panels/grafana-polystat-panel’, ‘mode’: ‘0755’, ‘isdir’: True, ‘ischr’: False, ‘isblk’: False, ‘isreg’: False, ‘isfifo’: False, ‘islnk’: False, ‘issock’: False, ‘uid’: 998, ‘gid’: 998, ‘size’: 4096, ‘inode’: 1677723979, ‘dev’: 49, ‘nlink’: 1, ‘atime’: 1709552496.0, ‘mtime’: 1715279282.4668195, ‘ctime’: 1715279282.4668195, ‘gr_name’: ‘grafana’, ‘pw_name’: ‘grafana’, ‘wusr’: True, ‘rusr’: True, ‘xusr’: True, ‘wgrp’: False, ‘rgrp’: True, ‘xgrp’: True, ‘woth’: False, ‘roth’: True, ‘xoth’: True, ‘isuid’: False, ‘isgid’: False})
changed: [localhost] => (item={‘path’: ‘/usr/share/percona-dashboards/panels/grafana-worldmap-panel’, ‘mode’: ‘0755’, ‘isdir’: True, ‘ischr’: False, ‘isblk’: False, ‘isreg’: False, ‘isfifo’: False, ‘islnk’: False, ‘issock’: False, ‘uid’: 998, ‘gid’: 998, ‘size’: 188, ‘inode’: 2013270222, ‘dev’: 49, ‘nlink’: 1, ‘atime’: 1709552496.0, ‘mtime’: 1715279282.4708197, ‘ctime’: 1715279282.4708197, ‘gr_name’: ‘grafana’, ‘pw_name’: ‘grafana’, ‘wusr’: True, ‘rusr’: True, ‘xusr’: True, ‘wgrp’: False, ‘rgrp’: True, ‘xgrp’: True, ‘woth’: False, ‘roth’: True, ‘xoth’: True, ‘isuid’: False, ‘isgid’: False})
changed: [localhost] => (item={‘path’: ‘/usr/share/percona-dashboards/panels/jdbranham-diagram-panel’, ‘mode’: ‘0755’, ‘isdir’: True, ‘ischr’: False, ‘isblk’: False, ‘isreg’: False, ‘isfifo’: False, ‘islnk’: False, ‘issock’: False, ‘uid’: 998, ‘gid’: 998, ‘size’: 175, ‘inode’: 201330282, ‘dev’: 49, ‘nlink’: 1, ‘atime’: 1709552496.0, ‘mtime’: 1715279282.4718196, ‘ctime’: 1715279282.4718196, ‘gr_name’: ‘grafana’, ‘pw_name’: ‘grafana’, ‘wusr’: True, ‘rusr’: True, ‘xusr’: True, ‘wgrp’: False, ‘rgrp’: True, ‘xgrp’: True, ‘woth’: False, ‘roth’: True, ‘xoth’: True, ‘isuid’: False, ‘isgid’: False})
changed: [localhost] => (item={‘path’: ‘/usr/share/percona-dashboards/panels/natel-discrete-panel’, ‘mode’: ‘0755’, ‘isdir’: True, ‘ischr’: False, ‘isblk’: False, ‘isreg’: False, ‘isfifo’: False, ‘islnk’: False, ‘issock’: False, ‘uid’: 998, ‘gid’: 998, ‘size’: 191, ‘inode’: 335547649, ‘dev’: 49, ‘nlink’: 1, ‘atime’: 1709552496.0, ‘mtime’: 1715279282.4738197, ‘ctime’: 1715279282.4738197, ‘gr_name’: ‘grafana’, ‘pw_name’: ‘grafana’, ‘wusr’: True, ‘rusr’: True, ‘xusr’: True, ‘wgrp’: False, ‘rgrp’: True, ‘xgrp’: True, ‘woth’: False, ‘roth’: True, ‘xoth’: True, ‘isuid’: False, ‘isgid’: False})
changed: [localhost] => (item={‘path’: ‘/usr/share/percona-dashboards/panels/petrslavotinek-carpetplot-panel’, ‘mode’: ‘0755’, ‘isdir’: True, ‘ischr’: False, ‘isblk’: False, ‘isreg’: False, ‘isfifo’: False, ‘islnk’: False, ‘issock’: False, ‘uid’: 998, ‘gid’: 998, ‘size’: 4096, ‘inode’: 536873352, ‘dev’: 49, ‘nlink’: 1, ‘atime’: 1709552496.0, ‘mtime’: 1715279282.4838197, ‘ctime’: 1715279282.4838197, ‘gr_name’: ‘grafana’, ‘pw_name’: ‘grafana’, ‘wusr’: True, ‘rusr’: True, ‘xusr’: True, ‘wgrp’: False, ‘rgrp’: True, ‘xgrp’: True, ‘woth’: False, ‘roth’: True, ‘xoth’: True, ‘isuid’: False, ‘isgid’: False})
changed: [localhost] => (item={‘path’: ‘/usr/share/percona-dashboards/panels/pmm-app’, ‘mode’: ‘0755’, ‘isdir’: True, ‘ischr’: False, ‘isblk’: False, ‘isreg’: False, ‘isfifo’: False, ‘islnk’: False, ‘issock’: False, ‘uid’: 998, ‘gid’: 998, ‘size’: 18, ‘inode’: 1342180922, ‘dev’: 49, ‘nlink’: 1, ‘atime’: 1709554108.0, ‘mtime’: 1709554108.0, ‘ctime’: 1715279282.2888186, ‘gr_name’: ‘grafana’, ‘pw_name’: ‘grafana’, ‘wusr’: True, ‘rusr’: True, ‘xusr’: True, ‘wgrp’: False, ‘rgrp’: True, ‘xgrp’: True, ‘woth’: False, ‘roth’: True, ‘xoth’: True, ‘isuid’: False, ‘isgid’: False})

TASK [dashboards_upgrade : Copy plugins to the plugin directory] ***************
changed: [localhost]

TASK [dashboards_upgrade : Set permissions for the plugin directory] ***********
changed: [localhost]

TASK [dashboards_upgrade : Check that the SQLite grafana database exists] ******
ok: [localhost]

TASK [dashboards_upgrade : Restart grafana before deleting data EL7] ***********
skipping: [localhost]

TASK [dashboards_upgrade : Restart grafana before deleting data EL9] ***********
changed: [localhost]

TASK [dashboards_upgrade : Remove old DBaaS dashboard (Before provisioning times)] ***
changed: [localhost]

TASK [dashboards_upgrade : Remove old PMM Inventory (Before provisioning times)] ***
changed: [localhost]

TASK [dashboards_upgrade : Remove old PMM Add Instance dashboard (Before provisioning times)] ***
changed: [localhost]

TASK [dashboards_upgrade : Remove old PMM Database Checks dashboard (Before provisioning times)] ***
changed: [localhost]

TASK [dashboards_upgrade : Remove old PMM Settings dashboard (Before provisioning times)] ***
changed: [localhost]

TASK [dashboards_upgrade : Remove the old clickhouse plugin] *******************
changed: [localhost]

TASK [dashboards_upgrade : Restart grafana with new plugins EL7] ***************
skipping: [localhost]

TASK [dashboards_upgrade : Restart grafana with new plugins EL9] ***************
changed: [localhost]

TASK [initialization : Create backup directory] ********************************
ok: [localhost]

TASK [initialization : Check if Postgres 11 exists] ****************************
ok: [localhost]

TASK [Upgrade Postgres database] ***********************************************
skipping: [localhost]

TASK [initialization : Create grafana database in postgres] ********************
ok: [localhost]

TASK [initialization : Create grafana user in postgres] ************************
[DEPRECATION WARNING]: Param ‘priv’ is deprecated. See the module docs for more
information. This feature will be removed from community.postgreql in version
4.0.0. Deprecation warnings can be disabled by setting
deprecation_warnings=False in ansible.cfg.
ok: [localhost]

TASK [initialization : Upgrade grafana database (Get the latest schema)] *******
skipping: [localhost]

TASK [initialization : Create working directory for Alertmanager] **************
ok: [localhost]

TASK [initialization : Create working directory for VictoriaMetrics] ***********
ok: [localhost]

TASK [initialization : Create empty configuration file for VictoriaMetrics] ****
changed: [localhost]

TASK [initialization : Check that the SQLite grafana database exists] **********
skipping: [localhost]

TASK [initialization : Remove database options (SQLite is default)] ************
skipping: [localhost]

TASK [initialization : Remove database host] ***********************************
skipping: [localhost]

TASK [initialization : Remove database user] ***********************************
skipping: [localhost]

TASK [initialization : Remove database password] *******************************
skipping: [localhost]

TASK [initialization : Upgrade grafana database (Get the latest schema)] *******
skipping: [localhost]

TASK [initialization : Start grafana again] ************************************
skipping: [localhost]

TASK [initialization : Wait for grafana] ***************************************
skipping: [localhost]

TASK [Migrate Grafana database from SQLite to Postgresql] **********************
skipping: [localhost]

TASK [initialization : Change default admin id] ********************************
skipping: [localhost]

TASK [initialization : Copy file with image version] ***************************
changed: [localhost]

TASK [initialization : Wait for PMM to be ready] *******************************
skipping: [localhost]

TASK [initialization : Disable maintenance mode] *******************************
skipping: [localhost]

TASK [Enable crond service] ****************************************************
skipping: [localhost]

TASK [Increase number of open files for jobs] **********************************
skipping: [localhost]

TASK [Reread supervisord configuration EL7] ************************************
skipping: [localhost]

TASK [Reread supervisord configuration EL9] ************************************
ok: [localhost]

TASK [Check reread results] ****************************************************
ok: [localhost] => {
“reread_result.stdout_lines”: [
“No config updates to processes”
]
}

TASK [Restart pmm-managed EL7] *************************************************
skipping: [localhost] => (item=stop)
skipping: [localhost] => (item=remove)
skipping: [localhost] => (item=add)
skipping: [localhost]

TASK [Restart pmm-managed EL9] *************************************************
changed: [localhost] => (item=stop)
changed: [localhost] => (item=remove)
changed: [localhost] => (item=add)

TASK [Wait for pmm-managed] ****************************************************
Pausing for 10 seconds
(ctrl+C then ‘C’ = continue early, ctrl+C then ‘A’ = abort)
ok: [localhost]

TASK [Update system packages EL7] **********************************************
skipping: [localhost]

TASK [Updating only select packages EL7] ***************************************
skipping: [localhost] => (item=nss)
skipping: [localhost] => (item=tzdata)
skipping: [localhost] => (item=libssh2)
skipping: [localhost] => (item=sshpass)
skipping: [localhost] => (item=vi)
skipping: [localhost]

TASK [Remove ansible RPM if installed | EL9] *********************************
ok: [localhost]

TASK [Install ansible-core RPM | EL9] *********************************
ok: [localhost]

TASK [Update system packages EL9] **********************************************
changed: [localhost]

TASK [Install nginx] ***********************************************************

TASK [nginx : Add Nginx repository for RHEL7] **********************************
skipping: [localhost]

TASK [nginx : Add Nginx repository for RHEL9] **********************************
ok: [localhost]

TASK [nginx : Create directories for nginx] ************************************
ok: [localhost] => (item=/usr/share/pmm-server/static/)
ok: [localhost] => (item=/etc/nginx/conf.d/)
ok: [localhost] => (item=/etc/nginx/ssl/)

TASK [nginx : Install nginx rpm | Install nginx] **********************
ok: [localhost]

TASK [nginx : Add ssl-related files and scripts] *******************************
ok: [localhost] => (item={‘src’: ‘ca-certs.pem’, ‘dest’: ‘/etc/nginx/ssl/ca-certs.pem’, ‘mode’: ‘u=rw,g=r,o=r’})
ok: [localhost] => (item={‘src’: ‘certificate.conf’, ‘dest’: ‘/etc/nginx/ssl/certificate.conf’, ‘mode’: ‘u=rw,g=r,o=r’})
ok: [localhost] => (item={‘src’: ‘dhparam.pem’, ‘dest’: ‘/etc/nginx/ssl/dhparam.pem’, ‘mode’: ‘u=rw,g=r,o=r’})
ok: [localhost] => (item={‘src’: ‘generate-ssl-certificate’, ‘dest’: ‘/var/lib/cloud/scripts/per-boot/generate-ssl-certificate’, ‘mode’: ‘u=rwx,g=rx,o=rx’})
ok: [localhost] => (item={‘src’: ‘upgrade-certificate’, ‘dest’: ‘/var/lib/cloud/scripts/per-boot/upgrade-certificate’, ‘mode’: ‘u=rwx,g=rx,o=rx’})

TASK [nginx : NGINX | Copy nginx configs] ****************
ok: [localhost] => (item=nginx.conf)
ok: [localhost] => (item=conf.d/pmm-ssl.conf)
ok: [localhost] => (item=conf.d/pmm.conf)

TASK [nginx : NGINX SSL Certificate | Check certificate file] *************
ok: [localhost]

TASK [nginx : NGINX SSL Certificate | Generate certificate] ***************
skipping: [localhost]

TASK [nginx : Remove default.conf] *********************************************
ok: [localhost]

TASK [nginx : Check nginx configuration] ***************************************
ok: [localhost]

TASK [nginx : Copy local-rss.xml file] *****************************************
ok: [localhost]

TASK [Install clickhouse] ******************************************************

TASK [clickhouse : Stop and remove clickhouse before update | EL7] *************
skipping: [localhost] => (item=stop)
skipping: [localhost] => (item=remove)
skipping: [localhost]

TASK [clickhouse : Find supervisord’s socket] **********************************
ok: [localhost]

TASK [clickhouse : Stop and remove clickhouse before update | EL9] *************
changed: [localhost] => (item=stop)
changed: [localhost] => (item=remove)

TASK [clickhouse : Remove old clickhouse packages] *****************************
changed: [localhost]

TASK [clickhouse : Disable clickhouse-server in systemd] ***********************
skipping: [localhost]

TASK [clickhouse : Add ClickHouse repository] **********************************
ok: [localhost]

TASK [clickhouse : Install clickhouse package] *********************************
changed: [localhost]

TASK [clickhouse : Copy clickhouse config to image] ****************************
changed: [localhost]

TASK [clickhouse : Remove cap_ipc_lock from clickhouse binary] *****************
ok: [localhost] => (item=cap_ipc_lock)
ok: [localhost] => (item=cap_sys_nice)

[Naresh9999]

ok: [localhost] => (item=cap_net_admin)

TASK [clickhouse : Remove clickhouse-odbc-bridge binary] ***********************
ok: [localhost]

TASK [clickhouse : Change ownership for clickhouse directory] ******************
changed: [localhost]

TASK [clickhouse : Start clickhouse EL9] ***************************************
changed: [localhost] => (item=present)
ok: [localhost] => (item=started)

TASK [clickhouse : Start clickhouse EL7] ***************************************
skipping: [localhost]

TASK [Copy rezise-xfs file for lvm] ********************************************
skipping: [localhost]

TASK [Create change-admin-password command] ************************************
ok: [localhost]

TASK [Check volume size] *******************************************************
skipping: [localhost]

TASK [Reread supervisord configuration again EL7] ******************************
skipping: [localhost]

TASK [Reread supervisord configuration again EL9] ******************************
ok: [localhost]

TASK [Check reread results] ****************************************************
ok: [localhost] => {
“reread_result.stdout_lines”: [
“No config updates to processes”
]
}

TASK [Restart services EL7] ****************************************************
skipping: [localhost] => (item=[‘alertmanager’, ‘stop’])
skipping: [localhost] => (item=[‘alertmanager’, ‘remove’])
skipping: [localhost] => (item=[‘alertmanager’, ‘add’])
skipping: [localhost] => (item=[‘nginx’, ‘stop’])
skipping: [localhost] => (item=[‘nginx’, ‘remove’])
skipping: [localhost] => (item=[‘nginx’, ‘add’])
skipping: [localhost] => (item=[‘grafana’, ‘stop’])
skipping: [localhost] => (item=[‘grafana’, ‘remove’])
skipping: [localhost] => (item=[‘grafana’, ‘add’])
skipping: [localhost] => (item=[‘qan-api2’, ‘stop’])
skipping: [localhost] => (item=[‘qan-api2’, ‘remove’])
skipping: [localhost] => (item=[‘qan-api2’, ‘add’])
skipping: [localhost] => (item=[‘pmm-agent’, ‘stop’])
skipping: [localhost] => (item=[‘pmm-agent’, ‘remove’])
skipping: [localhost] => (item=[‘pmm-agent’, ‘add’])
skipping: [localhost]

TASK [Restart services EL9] ****************************************************
changed: [localhost] => (item=[‘alertmanager’, ‘stop’])
changed: [localhost] => (item=[‘alertmanager’, ‘remove’])
changed: [localhost] => (item=[‘alertmanager’, ‘add’])
changed: [localhost] => (item=[‘nginx’, ‘stop’])
changed: [localhost] => (item=[‘nginx’, ‘remove’])
changed: [localhost] => (item=[‘nginx’, ‘add’])
changed: [localhost] => (item=[‘grafana’, ‘stop’])
changed: [localhost] => (item=[‘grafana’, ‘remove’])
changed: [localhost] => (item=[‘grafana’, ‘add’])
changed: [localhost] => (item=[‘qan-api2’, ‘stop’])
changed: [localhost] => (item=[‘qan-api2’, ‘remove’])
changed: [localhost] => (item=[‘qan-api2’, ‘add’])
changed: [localhost] => (item=[‘pmm-agent’, ‘stop’])
changed: [localhost] => (item=[‘pmm-agent’, ‘remove’])
changed: [localhost] => (item=[‘pmm-agent’, ‘add’])

TASK [Delete old grafana and yum-cron artifacts] *******************************
ok: [localhost] => (item=/var/lib/grafana/plugins/)
changed: [localhost] => (item=/usr/share/grafana/data/grafana.db)
ok: [localhost] => (item=/etc/yum/yum-cron-hourly.conf)
ok: [localhost] => (item=/etc/yum/yum-cron.conf)

TASK [Check that the SQLite grafana database exists] ***************************
ok: [localhost]

TASK [Migrate Grafana database from SQLite to Postgresql] **********************

TASK [sqlite-to-postgres : Wait for grafana to update DB and be ready] *********
ok: [localhost]

TASK [sqlite-to-postgres : Create Grafana backup dir] **************************
ok: [localhost]

TASK [sqlite-to-postgres : Stop grafana before upgrade] ************************
changed: [localhost]

TASK [sqlite-to-postgres : Drop grafana database from postgres EL7] ************
skipping: [localhost]

TASK [sqlite-to-postgres : Drop grafana database from postgres EL9] ************
changed: [localhost]

TASK [sqlite-to-postgres : Create grafana database in postgres] ****************
changed: [localhost]

TASK [sqlite-to-postgres : Create grafana user in postgres] ********************
changed: [localhost]

TASK [sqlite-to-postgres : Create backup for SQLite Grafana database] **********
changed: [localhost]

TASK [sqlite-to-postgres : Remove all ` symbols in grafana dashboard description] ***
changed: [localhost]

TASK [sqlite-to-postgres : Disable provisioning before change database] ********
changed: [localhost]

TASK [sqlite-to-postgres : Switch to postgres] *********************************
changed: [localhost]

TASK [sqlite-to-postgres : Set database host] **********************************
changed: [localhost]

TASK [sqlite-to-postgres : Set database user] **********************************
changed: [localhost]

TASK [sqlite-to-postgres : Set database password] ******************************
changed: [localhost]

TASK [sqlite-to-postgres : Start grafana again] ********************************
changed: [localhost]

TASK [sqlite-to-postgres : Check if initial data were created] *****************
ok: [localhost]

TASK [sqlite-to-postgres : Wait for grafana database initialization] ***********
Pausing for 10 seconds
(ctrl+C then ‘C’ = continue early, ctrl+C then ‘A’ = abort)
ok: [localhost]

TASK [sqlite-to-postgres : Stop grafana before upgrade] ************************
changed: [localhost]

TASK [sqlite-to-postgres : Remove default admin user] **************************
changed: [localhost]

TASK [sqlite-to-postgres : Run grafana migrator] *******************************
ok: [localhost]

TASK [sqlite-to-postgres : copy the output of grafana-db-migrator to file] *****
ok: [localhost]

TASK [sqlite-to-postgres : Enable provisioning after change database] **********
changed: [localhost]

TASK [sqlite-to-postgres : Start grafana again] ********************************
changed: [localhost]

TASK [sqlite-to-postgres : Wait for grafana initialization] ********************
Pausing for 5 seconds
(ctrl+C then ‘C’ = continue early, ctrl+C then ‘A’ = abort)
ok: [localhost]

TASK [sqlite-to-postgres : Fix database/folder relationship] *******************
ok: [localhost]

TASK [sqlite-to-postgres : Copy the output of grafana-db-migrator to file] *****
ok: [localhost]

TASK [sqlite-to-postgres : Remove SQLite Grafana database] *********************
changed: [localhost]

TASK [Fix grafana fields type] *************************************************
changed: [localhost] => (item=ALTER TABLE tag ALTER COLUMN key TYPE text;)
changed: [localhost] => (item=ALTER TABLE tag ALTER COLUMN value TYPE text;)
changed: [localhost] => (item=ALTER TABLE api_key ALTER COLUMN key TYPE text;)
changed: [localhost] => (item=ALTER TABLE api_key ALTER COLUMN name TYPE text;)

TASK [Change default admin id] *************************************************
changed: [localhost]

TASK [Remove old or redundant packages] ****************************************
ok: [localhost]

TASK [Generate new supervisor config] ******************************************
changed: [localhost]

TASK [Reread pmm-update-perform-init supervisor config EL7] ********************
skipping: [localhost]

TASK [Reread pmm-update-perform-init supervisor config EL9] ********************
changed: [localhost]

TASK [Restart pmm-managed EL7] *************************************************
skipping: [localhost] => (item=stop)
skipping: [localhost] => (item=remove)
skipping: [localhost] => (item=add)
skipping: [localhost]

TASK [Restart pmm-managed EL9] *************************************************
changed: [localhost] => (item=stop)
changed: [localhost] => (item=remove)
changed: [localhost] => (item=add)

TASK [Update/restart other services EL7] ***************************************
skipping: [localhost]

TASK [Update/restart other services EL9] ***************************************
changed: [localhost]

TASK [Print other services’s logs] *********************************************
ok: [localhost] => {
“update_result.stdout_lines”: [
“pmm-update-perform-init: stopped”,
“pmm-update-perform-init: updated process group”
]
}

TASK [Wait for PMM to be ready] ************************************************
FAILED - RETRYING: [localhost]: Wait for PMM to be ready (120 retries left).
ok: [localhost]

TASK [Get supervisord logs EL7] ************************************************
skipping: [localhost]

TASK [Get supervisord logs EL9] ************************************************
ok: [localhost]

TASK [Print supervisord logs] **************************************************
ok: [localhost] => {
“maintail_result.stdout_lines”: [
“2024-05-09 18:27:31,210 INFO received SIGUSR2 indicating log reopen request”,
“2024-05-09 18:27:31,211 INFO supervisord logreopen”,
“2024-05-09 18:27:31,452 INFO spawned: ‘pmm-update-perform’ with pid 3883406”,
“2024-05-09 18:27:32,448 INFO success: pmm-update-perform entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)”,
“2024-05-09 18:27:35,307 INFO exited: pmm-update-perform (exit status 42; not expected)”,
“2024-05-09 18:27:35,311 INFO spawned: ‘pmm-update-perform’ with pid 3883548”,
“2024-05-09 18:27:36,316 INFO success: pmm-update-perform entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)”,
“2024-05-09 18:29:37,114 INFO waiting for clickhouse to stop”,
“2024-05-09 18:29:39,114 INFO waiting for clickhouse to stop”,
“2024-05-09 18:29:41,114 INFO waiting for clickhouse to stop”,
“2024-05-09 18:29:43,114 INFO waiting for clickhouse to stop”,
“2024-05-09 18:29:43,726 INFO stopped: clickhouse (exit status 0)”,
“2024-05-09 18:29:43,730 INFO spawned: ‘clickhouse’ with pid 3888375”,
“2024-05-09 18:29:44,730 INFO success: clickhouse entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)”,
“2024-05-09 18:29:50,865 INFO waiting for grafana to stop”,
“2024-05-09 18:29:50,967 INFO stopped: grafana (exit status 0)”,
“2024-05-09 18:29:53,950 INFO spawned: ‘grafana’ with pid 3889360”,
“2024-05-09 18:29:54,953 INFO success: grafana entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)”,
“2024-05-09 18:29:59,093 INFO waiting for pmm-managed to stop”,
“2024-05-09 18:29:59,204 INFO stopped: pmm-managed (exit status 0)”,
“2024-05-09 18:29:59,204 INFO reaped unknown pid 3889787 (terminated by SIGKILL)”,
“2024-05-09 18:30:00,090 INFO spawned: ‘pmm-managed’ with pid 3889819”,
“2024-05-09 18:30:01,091 INFO success: pmm-managed entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)”,
“2024-05-09 18:31:36,003 INFO waiting for clickhouse to stop”,
“2024-05-09 18:31:38,003 INFO waiting for clickhouse to stop”,
“2024-05-09 18:31:40,003 INFO waiting for clickhouse to stop”,
“2024-05-09 18:31:41,974 INFO stopped: clickhouse (exit status 0)”,
“2024-05-09 18:31:46,113 INFO reaped unknown pid 3891279 (exit status 0)”,
“2024-05-09 18:31:46,149 INFO reaped unknown pid 3891281 (exit status 0)”,
“2024-05-09 18:31:46,149 INFO reaped unknown pid 3891283 (exit status 0)”,
“2024-05-09 18:31:46,149 INFO reaped unknown pid 3891285 (exit status 8)”,
“2024-05-09 18:31:46,149 INFO reaped unknown pid 3891287 (exit status 0)”,
“2024-05-09 18:31:46,155 INFO reaped unknown pid 3891289 (exit status 0)”,
“2024-05-09 18:31:46,219 INFO reaped unknown pid 3891291 (exit status 0)”,
“2024-05-09 18:31:47,431 INFO reaped unknown pid 3891298 (exit status 0)”,
“2024-05-09 18:31:47,452 INFO reaped unknown pid 3891300 (exit status 0)”,
“2024-05-09 18:31:52,950 INFO spawned: ‘clickhouse’ with pid 3891462”,
“2024-05-09 18:31:53,951 INFO success: clickhouse entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)”,
“2024-05-09 18:31:55,059 INFO waiting for alertmanager to stop”,
“2024-05-09 18:31:55,083 INFO stopped: alertmanager (exit status 0)”,
“2024-05-09 18:31:56,003 INFO spawned: ‘alertmanager’ with pid 3891596”,
“2024-05-09 18:31:56,503 INFO waiting for nginx to stop”,
“2024-05-09 18:31:56,616 INFO stopped: nginx (exit status 0)”,
“2024-05-09 18:31:57,006 INFO success: alertmanager entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)”,
“2024-05-09 18:31:57,586 INFO spawned: ‘nginx’ with pid 3892033”,
“2024-05-09 18:31:58,055 INFO waiting for grafana to stop”,
“2024-05-09 18:31:58,068 INFO stopped: grafana (exit status 0)”,
“2024-05-09 18:31:58,630 INFO success: nginx entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)”,
“2024-05-09 18:31:58,943 INFO spawned: ‘grafana’ with pid 3892095”,
“2024-05-09 18:31:59,391 INFO waiting for qan-api2 to stop”,
“2024-05-09 18:31:59,455 INFO stopped: qan-api2 (exit status 0)”,
“2024-05-09 18:31:59,960 INFO success: grafana entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)”,
“2024-05-09 18:32:00,298 INFO spawned: ‘qan-api2’ with pid 3892211”,
“2024-05-09 18:32:00,724 INFO waiting for pmm-agent to stop”,
“2024-05-09 18:32:00,742 INFO stopped: pmm-agent (exit status 0)”,
“2024-05-09 18:32:01,326 INFO success: qan-api2 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)”,
“2024-05-09 18:32:04,712 INFO waiting for grafana to stop”,
“2024-05-09 18:32:04,727 INFO stopped: grafana (exit status 0)”,
“2024-05-09 18:32:09,171 INFO spawned: ‘grafana’ with pid 3892627”,
“2024-05-09 18:32:10,170 INFO success: grafana entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)”,
“2024-05-09 18:32:21,302 INFO waiting for grafana to stop”,
“2024-05-09 18:32:21,313 INFO stopped: grafana (exit status 0)”,
“2024-05-09 18:32:26,753 INFO spawned: ‘grafana’ with pid 3892951”,
“2024-05-09 18:32:27,756 INFO success: grafana entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)”,
“2024-05-09 18:32:37,995 INFO waiting for pmm-managed to stop”,
“2024-05-09 18:32:38,009 INFO stopped: pmm-managed (exit status 0)”,
“2024-05-09 18:32:38,009 INFO reaped unknown pid 3893307 (terminated by SIGKILL)”,
“2024-05-09 18:32:38,865 INFO spawned: ‘pmm-managed’ with pid 3893341”,
“2024-05-09 18:32:39,344 INFO spawned: ‘pmm-agent’ with pid 3893418”,
“2024-05-09 18:32:39,361 INFO spawned: ‘pmm-update-perform-init’ with pid 3893419”,
“2024-05-09 18:32:39,870 INFO success: pmm-managed entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)”,
“2024-05-09 18:32:40,357 INFO success: pmm-agent entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)”,
“2024-05-09 18:32:40,360 INFO success: pmm-update-perform-init entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)”
]
}

TASK [Disable maintenance mode] ************************************************
changed: [localhost]

TASK [Cleanup yum cache] *******************************************************
ok: [localhost]

PLAY RECAP *********************************************************************
localhost : ok=140 changed=66 unreachable=0 failed=0 skipped=79 rescued=0 ignored=0

time=“2024-05-09T18:32:42Z” level=info msg=“Waiting for Grafana dashboards update to finish…”

[Naresh9999]

Can someone please help me with the above issue?

[Naresh9999]

@nurlan Can you please help me on the above issue?

[rups]

Hi ,

I am facing similar issue after upgrade to 2.41.2, my dashboard wont come up. Upgrade failed and now I am stuck with internal server error. Is there a resolution.

[Naresh9999]

Hi @rups

Can you please check the upgrade logs or grafana logs?

[rups]

Hi @Naresh9999

Sharing log snippets from pmm docker container. I am not sure why it errored out as permission denied.

From Grafana logs:/srv/logs/grafana.log
logger=settings t=2024-06-17T22:20:26.381311466Z level=info msg=“Starting Grafana” version=9.2.20 commit=9.2.20 branch=main compiled=2023-06-27T00:00:00Z
logger=settings t=2024-06-17T22:20:26.381507239Z level=info msg=“Config loaded from” file=/usr/share/grafana/conf/defaults.ini
logger=settings t=2024-06-17T22:20:26.38151564Z level=info msg=“Config loaded from” file=/etc/grafana/grafana.ini
logger=settings t=2024-06-17T22:20:26.381519001Z level=info msg=“Path Home” path=/usr/share/grafana
logger=settings t=2024-06-17T22:20:26.381522032Z level=info msg=“Path Data” path=/srv/grafana
logger=settings t=2024-06-17T22:20:26.381524725Z level=info msg=“Path Logs” path=/srv/logs
logger=settings t=2024-06-17T22:20:26.381527481Z level=info msg=“Path Plugins” path=/srv/grafana/plugins
logger=settings t=2024-06-17T22:20:26.381530132Z level=info msg=“Path Provisioning” path=/usr/share/grafana/conf/provisioning
logger=settings t=2024-06-17T22:20:26.381533498Z level=info msg=“App mode production”
logger=sqlstore t=2024-06-17T22:20:26.381600948Z level=info msg=“Connecting to DB” dbtype=sqlite3
logger=migrator t=2024-06-17T22:20:26.382166232Z level=error msg=“alert migration failure: could not get migration log” error=“failed to check table existence: unable to open database file: permission denied”

Upgrade logs
TASK [Check that the SQLite grafana database exists] ***************************
ok: [localhost]
TASK [Migrate Grafana database from SQLite to Postgresql] **********************
TASK [sqlite-to-postgres : Wait for grafana to update DB and be ready] *********
FAILED - RETRYING: [localhost]: Wait for grafana to update DB and be ready (120 retries left).
FAILED - RETRYING: [localhost]: Wait for grafana to update DB and be ready (43 retries left).
FAILED - RETRYING: [localhost]: Wait for grafana to update DB and be ready (42 retries left).
FAILED - RETRYING: [localhost]: Wait for grafana to update DB and be ready (41 retries left).
FAILED - RETRYING: [localhost]: Wait for grafana to update DB and be ready (40 retries left).
FAILED - RETRYING: [localhost]: Wait for grafana to update DB and be ready (39 retries left).
FAILED - RETRYING: [localhost]: Wait for grafana to update DB and be ready (38 retries left).
FAILED - RETRYING: [localhost]: Wait for grafana to update DB and be ready (37 retries left).
FAILED - RETRYING: [localhost]: Wait for grafana to update DB and be ready (36 retries left).
FAILED - RETRYING: [localhost]: Wait for grafana to update DB and be ready (35 retries left).

[Naresh9999]

Hi @rups

This seems to be the same issue which I faced, Please go through the below ticket, so that you will get some clarity.

However, from what I understand, we should not use the old grafana.ini file. After upgrading, we must use the latest grafana.ini file and add the appropriate configurations to the most recent grafana.ini file. Whenever you perform an upgrade, you must update the grafana.ini file instead of replacing it with an old grafana.ini file.

[PMM-13114] After upgrading from 2.41.1 to 2.41.2 PostgreSQL DB (Grafana DB) is empty inside PMM server container - Percona JIRA (atlassian.net)