PMM3 | PostgreSQL | Inside container FATAL: role does not exist

Percona Monitoring and Management (PMM)
1

Description:

I made a backup of ClickhouseDB, VM, and PostgreSQL after updating from 2.44 to 3.2.0.

On the new testing server, I installed the 3.2.0 version and restored the backup to see if it was functioning properly. I am unable to view any PostgreSQL-related metrics on the PMM Health Check Dashboard following the backup import.

image
image1919×736 101 KB

image
image1482×615 47.2 KB

image
image1481×684 42 KB

Steps to Reproduce:

  1. Upgrade PMM version from 2.44.0 to 3.2.0
  2. Take a backup after the 3.2.0 upgrade.
  3. Install 3.2.0 on the new test server
  4. Restore the backup to the new test server.
  5. Verify the PostgreSQL dashboards in the PMM Health. Here metrics are empty.

Version:

PMM - 3.2.0

Logs:

Inside the PMM3.2.0 container, observed the below errors.

From the postgresql14.log:

postgres=#
2025-06-24 07:38:42.710 UTC [35048] FATAL: role “AX4rfqpWrkZZADKTxBeIh9AZDlLlNXPtvMgznavIl82Vc2U5+LZLX+P1XG8=” does not exist
2025-06-24 07:38:43.991 UTC [35050] FATAL: role “AX4rfqpWrkZZADKTxBeIh9AZDlLlNXPtvMgznavIl82Vc2U5+LZLX+P1XG8=” does not exist
2025-06-24 07:38:43.992 UTC [35051] FATAL: role “AX4rfqpWrkZZADKTxBeIh9AZDlLlNXPtvMgznavIl82Vc2U5+LZLX+P1XG8=” does not exist
2025-06-24 07:38:43.993 UTC [35052] FATAL: role “AX4rfqpWrkZZADKTxBeIh9AZDlLlNXPtvMgznavIl82Vc2U5+LZLX+P1XG8=” does not exist
2025-06-24 07:38:44.996 UTC [35053] FATAL: role “AX4rfqpWrkZZADKTxBeIh9AZDlLlNXPtvMgznavIl82Vc2U5+LZLX+P1XG8=” does not exist
2025-06-24 07:38:44.996 UTC [35054] FATAL: role “AX4rfqpWrkZZADKTxBeIh9AZDlLlNXPtvMgznavIl82Vc2U5+LZLX+P1XG8=” does not exist
2025-06-24 07:38:48.992 UTC [35067] FATAL: role “AX4rfqpWrkZZADKTxBeIh9AZDlLlNXPtvMgznavIl82Vc2U5+LZLX+P1XG8=” does not exist
2025-06-24 07:38:48.992 UTC [35068] FATAL: role “AX4rfqpWrkZZADKTxBeIh9AZDlLlNXPtvMgznavIl82Vc2U5+LZLX+P1XG8=” does not exist
2025-06-24 07:38:48.993 UTC [35069] FATAL: role “AX4rfqpWrkZZADKTxBeIh9AZDlLlNXPtvMgznavIl82Vc2U5+LZLX+P1XG8=” does not exist
2025-06-24 07:38:49.997 UTC [35070] FATAL: role “AX4rfqpWrkZZADKTxBeIh9AZDlLlNXPtvMgznavIl82Vc2U5+LZLX+P1XG8=” does not exist
2025-06-24 07:38:49.997 UTC [35071] FATAL: role “AX4rfqpWrkZZADKTxBeIh9AZDlLlNXPtvMgznavIl82Vc2U5+LZLX+P1XG8=” does not exist
2025-06-24 07:38:51.704 UTC [35078] FATAL: role “AX4rfqpWrkZZADKTxBeIh9AZDlLlNXPtvMgznavIl82Vc2U5+LZLX+P1XG8=” does not exist

From the pmm-agent.log:

time=“2025-06-24T08:52:48.989+00:00” level=info msg=“ts=2025-06-24T08:52:48.988Z caller=server.go:75 level=info msg="Established new database connection" fingerprint=127.0.0.1:5432” agentID=be09d7c1-6b62-4c0a-b1d7-19f2af019646 component=agent-process type=postgres_exporter
time=“2025-06-24T08:52:48.994+00:00” level=error msg=“ts=2025-06-24T08:52:48.993Z caller=collector.go:202 level=error msg="Error opening connection to database" err="error querying postgresql version: pq: role \"AX4rfqpWrkZZADKTxBeIh9AZDlLlNXPtvMgznavIl82Vc2U5+LZLX+P1XG8=\" does not exist"” agentID=be09d7c1-6b62-4c0a-b1d7-19f2af019646 component=agent-process type=postgres_exporter
time=“2025-06-24T08:52:49.994+00:00” level=error msg=“ts=2025-06-24T08:52:49.994Z caller=datasource.go:107 level=error msg="Error opening connection to database" dsn="postgres://AX4rfqpWrkZZADKTxBeIh9AZDlLlNXPtvMgznavIl82Vc2U5+LZLX+P1XG8=:PASSWORD_REMOVED@127.0.0.1:5432/postgres?connect_timeout=1&sslmode=disable" err="pq: role \"AX4rfqpWrkZZADKTxBeIh9AZDlLlNXPtvMgznavIl82Vc2U5+LZLX+P1XG8=\" does not exist"” agentID=be09d7c1-6b62-4c0a-b1d7-19f2af019646 component=agent-process type=postgres_exporter
time=“2025-06-24T08:52:49.994+00:00” level=info msg=“ts=2025-06-24T08:52:49.994Z caller=server.go:75 level=info msg="Established new database connection" fingerprint=127.0.0.1:5432” agentID=be09d7c1-6b62-4c0a-b1d7-19f2af019646 component=agent-process type=postgres_exporter
time=“2025-06-24T08:52:49.995+00:00” level=error msg=“ts=2025-06-24T08:52:49.994Z caller=datasource.go:107 level=error msg="Error opening connection to database" dsn="postgres://AX4rfqpWrkZZADKTxBeIh9AZDlLlNXPtvMgznavIl82Vc2U5+LZLX+P1XG8=:PASSWORD_REMOVED@127.0.0.1:5432/postgres?connect_timeout=1&sslmode=disable" err="pq: role \"AX4rfqpWrkZZADKTxBeIh9AZDlLlNXPtvMgznavIl82Vc2U5+LZLX+P1XG8=\" does not exist"” agentID=be09d7c1-6b62-4c0a-b1d7-19f2af019646 component=agent-process type=postgres_exporter
time=“2025-06-24T08:52:49.995+00:00” level=info msg=“ts=2025-06-24T08:52:49.994Z caller=server.go:75 level=info msg="Established new database connection" fingerprint=127.0.0.1:5432” agentID=be09d7c1-6b62-4c0a-b1d7-19f2af019646 component=agent-process type=postgres_exporter
time=“2025-06-24T08:52:50.999+00:00” level=error msg=“ts=2025-06-24T08:52:50.998Z caller=postgres_exporter.go:770 level=error err="Error opening connection to database (postgres://AX4rfqpWrkZZADKTxBeIh9AZDlLlNXPtvMgznavIl82Vc2U5+LZLX+P1XG8=:PASSWORD_REMOVED@127.0.0.1:5432/postgres?connect_timeout=1&sslmode=disable): pq: role \"AX4rfqpWrkZZADKTxBeIh9AZDlLlNXPtvMgznavIl82Vc2U5+LZLX+P1XG8=\" does not exist"” agentID=be09d7c1-6b62-4c0a-b1d7-19f2af019646 component=agent-process type=postgres_exporter
time=“2025-06-24T08:52:50.999+00:00” level=error msg=“ts=2025-06-24T08:52:50.999Z caller=postgres_exporter.go:770 level=error err="Error opening connection to database (postgres://AX4rfqpWrkZZADKTxBeIh9AZDlLlNXPtvMgznavIl82Vc2U5+LZLX+P1XG8=:PASSWORD_REMOVED@127.0.0.1:5432/postgres?connect_timeout=1&sslmode=disable): pq: role \"AX4rfqpWrkZZADKTxBeIh9AZDlLlNXPtvMgznavIl82Vc2U5+LZLX+P1XG8=\" does not exist"” agentID=be09d7c1-6b62-4c0a-b1d7-19f2af019646 component=agent-process type=postgres_exporter
time=“2025-06-24T08:52:51.702+00:00” level=info msg=“ts=2025-06-24T08:52:51.702Z caller=percona_exporter.go:86 msg="Excluded databases" databases="[template0 template1 cloudsqladmin pmm-managed-dev azure_maintenance rdsadmin]"” agentID=be09d7c1-6b62-4c0a-b1d7-19f2af019646 component=agent-process type=postgres_exporter
time=“2025-06-24T08:52:51.702+00:00” level=info msg=“ts=2025-06-24T08:52:51.702Z caller=server.go:75 level=info msg="Established new database connection" fingerprint=127.0.0.1:5432” agentID=be09d7c1-6b62-4c0a-b1d7-19f2af019646 component=agent-process type=postgres_exporter
time=“2025-06-24T08:52:52.159+00:00” level=error msg=“cannot get MySQL version: Error 1045 (28000): Access denied for user ‘AX4rfqpFN9wEjDQhoUufeTDLA3QFWlOQ0k9ZmVK3p8vWLFHG’@‘101.11.8.17’ (using password: YES)” agentID=6237ca76-efc0-4db9-9b0a-77b726fdfa21 component=agent-builtin type=qan_mysql_perfschema_agent
time=“2025-06-24T08:52:52.511+00:00” level=info msg=“time=2025-06-24T08:52:52.510Z level=INFO source=mysqld_exporter.go:243 msg=collect params=custom_query.hr,global_status,info_schema.innodb_metrics,standard.go,standard.process” agentID=75ee6d78-0749-4fb0-852e-b1e943e101e4 component=agent-process type=mysqld_exporter
time=“2025-06-24T08:52:52.539+00:00” level=error msg=“time=2025-06-24T08:52:52.539Z level=ERROR source=exporter.go:137 msg="Error opening connection to database" err="Error 1045 (28000): Access denied for user ‘AX4rfqoj4flI70EYXybEL56vKOERIPVZAQBpQHnOJxcpVwZ+’@‘101.11.8.17’ (using password: YES)"” agentID=75ee6d78-0749-4fb0-852e-b1e943e101e4 component=agent-process type=mysqld_exporter
time=“2025-06-24T08:52:52.707+00:00” level=error msg=“ts=2025-06-24T08:52:52.706Z caller=datasource.go:107 level=error msg="Error opening connection to database" dsn="postgres://AX4rfqpWrkZZADKTxBeIh9AZDlLlNXPtvMgznavIl82Vc2U5+LZLX+P1XG8=:PASSWORD_REMOVED@127.0.0.1:5432/postgres?connect_timeout=1&sslmode=disable" err="pq: role \"AX4rfqpWrkZZADKTxBeIh9AZDlLlNXPtvMgznavIl82Vc2U5+LZLX+P1XG8=\" does not exist"” agentID=be09d7c1-6b62-4c0a-b1d7-19f2af019646 component=agent-process type=postgres_exporter
time=“2025-06-24T08:52:52.707+00:00” level=info msg=“ts=2025-06-24T08:52:52.707Z caller=server.go:75 level=info msg="Established new database connection" fingerprint=127.0.0.1:5432” agentID=be09d7c1-6b62-4c0a-b1d7-19f2af019646 component=agent-process type=postgres_exporter

2

PostgreSQL restore went fine, but after that only Qan postgresql pgstatements agent is failing with the above errors.

3

The solution is provided in the below forum link: