# PMM3 postgres on Read only =\> pg\_custom\_replication\_wal pq: recovery is in progress **URL:** \ **Category:** PMM 3.x\ **Created:** [July 15, 2025, 9:18am UTC](https://forums.percona.com/t/pmm3-postgres-on-read-only-pg-custom-replication-wal-pq-recovery-is-in-progress/38681 "2025-07-15T09:18:05Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![bart.schelstraete](https://avatars.discourse-cdn.com/v4/letter/b/82dd89/32.png) [@bart.schelstraete](https://forums.percona.com/u/bart.schelstraete)\ **Post date:** [July 15, 2025, 9:18am UTC](https://forums.percona.com/t/pmm3-postgres-on-read-only-pg-custom-replication-wal-pq-recovery-is-in-progress/38681/1 "2025-07-15T09:18:05Z")
Since the last version, our syslog is being spammed on read only postgres **slaves** : Jul 15 10:15:59 pmm-agent[2743907]: time=“2025-07-15T10:15:59.975+01:00” level=error msg=“ts=2025-07-15T09:15:59.974Z caller=server.go:132 level=error namespace=pg\_custom\_replication\_wal msg="Error running query on database \"xx:5432\": pg\_custom\_replication\_wal pq: recovery is in progress"” agentID=xxx-e4ec-4baa-8c05-0c24573f62b9 component=agent-process type=postgres\_exporter Jul 15 10:15:59 pmm-agent[2743907]: time=“2025-07-15T10:15:59.975+01:00” level=error msg=“ts=2025-07-15T09:15:59.974Z caller=postgres\_exporter.go:770 level=error err="queryNamespaceMappings returned 1 errors"” agentID=xxx-e4ec-4baa-8c05-0c24573f62b9 component=agent-process type=postgres\_exporter ---
**Author:** ![anil.joshi](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/anil.joshi/32/13100_2.png) [@anil.joshi](https://forums.percona.com/u/anil.joshi)\ **Post date:** [July 30, 2025, 8:14am UTC](https://forums.percona.com/t/pmm3-postgres-on-read-only-pg-custom-replication-wal-pq-recovery-is-in-progress/38681/2 "2025-07-30T08:14:42Z")
@bart.schelstraete It looks like you are running some custom metric collector:- **pg\_custom\_replication\_wal** , which is unable to perform operations on read-only replicas. Do you have the details on what exactly the custom metrics are trying to get or execute? If I am correct, you are not seeing these messages in the Primary. Right? ---
**Author:** ![Andrin](https://avatars.discourse-cdn.com/v4/letter/a/8e7dd6/32.png) [@Andrin](https://forums.percona.com/u/Andrin)\ **Post date:** [August 26, 2025, 2:59pm UTC](https://forums.percona.com/t/pmm3-postgres-on-read-only-pg-custom-replication-wal-pq-recovery-is-in-progress/38681/3 "2025-08-26T14:59:28Z")
I’ve run into the same issue, described more in detail in this [github issue](https://github.com/percona/percona-postgresql-operator/issues/1259). There is monitoring in my kluster already present, but it does not autodetect anything and is also not configured to scrape from these instances. So I don’t think it has anything to do with custom queries, but rather with the configuration of pmm-client that natively ships with operator 2.7.0. Or, as I assume, it has something to do with the postgres\_exporter and its inability to skip querying instances that are in replica state. ---
**Author:** ![deskr](https://avatars.discourse-cdn.com/v4/letter/d/bbce88/32.png) [@deskr](https://forums.percona.com/u/deskr)\ **Post date:** [September 2, 2025, 7:35am UTC](https://forums.percona.com/t/pmm3-postgres-on-read-only-pg-custom-replication-wal-pq-recovery-is-in-progress/38681/4 "2025-09-02T07:35:50Z")
I have same issues on the slaves. Is there any solutions for slave databases? What’s the best practise for postgres clusters with PMM? ---
**Author:** ![Bart\_Schelstraete](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/bart_schelstraete/32/21033_2.png) [@Bart\_Schelstraete](https://forums.percona.com/u/Bart_Schelstraete)\ **Post date:** [November 18, 2025, 9:22pm UTC](https://forums.percona.com/t/pmm3-postgres-on-read-only-pg-custom-replication-wal-pq-recovery-is-in-progress/38681/5 "2025-11-18T21:22:23Z")
this is no custom metric collector, this is default monitoring. It used to work without any problem in previous version too (3.2). ---
**Author:** ![ulka01](https://avatars.discourse-cdn.com/v4/letter/u/77aa72/32.png) [@ulka01](https://forums.percona.com/u/ulka01)\ **Post date:** [December 11, 2025, 9:34am UTC](https://forums.percona.com/t/pmm3-postgres-on-read-only-pg-custom-replication-wal-pq-recovery-is-in-progress/38681/6 "2025-12-11T09:34:02Z")
I have same issues in 3.4.1 ---
**Author:** ![deskr](https://avatars.discourse-cdn.com/v4/letter/d/bbce88/32.png) [@deskr](https://forums.percona.com/u/deskr)\ **Post date:** [December 11, 2025, 9:45am UTC](https://forums.percona.com/t/pmm3-postgres-on-read-only-pg-custom-replication-wal-pq-recovery-is-in-progress/38681/7 "2025-12-11T09:45:45Z")
Fixed in version 3.5.0! > > > \### Report > > After setting up a postgres cluster with percona-postgres-operator a…nd enabling pmm-client on it, the pmm-client sidecars that are attached to replica postgres instances spam the following errors: > > \`\`\` > time="2025-08-25T15:29:02.407+00:00" level=info msg="ts=2025-08-25T15:29:02.407Z caller=percona\_exporter.go:86 msg=\\"Excluded databases\\" databases=\\"\[template0 template1 cloudsqladmin pmm-managed-dev azure\_maintenance rdsadmin\]\\"" agentID=cd482abd-7606-4364-a30b-3105066e76e7 component=agent-process type=postgres\_exporter > time="2025-08-25T15:29:07.094+00:00" level=info msg="ts=2025-08-25T15:29:07.094Z caller=percona\_exporter.go:86 msg=\\"Excluded databases\\" databases=\\"\[template0 template1 cloudsqladmin pmm-managed-dev azure\_maintenance rdsadmin\]\\"" agentID=cd482abd-7606-4364-a30b-3105066e76e7 component=agent-process type=postgres\_exporter > time="2025-08-25T15:29:07.114+00:00" level=error msg="ts=2025-08-25T15:29:07.113Z caller=namespace.go:236 level=error err=\\"Error running query on database \\\\\\"127.0.0.1:5432\\\\\\": pg\_custom\_replication\_wal pq: recovery is in progress\\"" agentID=cd482abd-7606-4364-a30b-3105066e76e7 component=agent-process type=postgres\_exporter > time="2025-08-25T15:29:07.117+00:00" level=error msg="ts=2025-08-25T15:29:07.117Z caller=namespace.go:236 level=error err=\\"Error running query on database \\\\\\"127.0.0.1:5432\\\\\\": pg\_custom\_replication\_wal pq: recovery is in progress\\"" agentID=cd482abd-7606-4364-a30b-3105066e76e7 component=agent-process type=postgres\_exporter > time="2025-08-25T15:29:07.119+00:00" level=error msg="ts=2025-08-25T15:29:07.119Z caller=server.go:130 level=error msg=\\"NAMESPACE ERRORS FOUND\\"" agentID=cd482abd-7606-4364-a30b-3105066e76e7 component=agent-process type=postgres\_exporter > time="2025-08-25T15:29:07.119+00:00" level=error msg="ts=2025-08-25T15:29:07.119Z caller=server.go:132 level=error namespace=pg\_custom\_replication\_wal msg=\\"Error running query on database \\\\\\"127.0.0.1:5432\\\\\\": pg\_custom\_replication\_wal pq: recovery is in progress\\"" agentID=cd482abd-7606-4364-a30b-3105066e76e7 component=agent-process type=postgres\_exporter > time="2025-08-25T15:29:07.119+00:00" level=error msg="ts=2025-08-25T15:29:07.119Z caller=postgres\_exporter.go:770 level=error err=\\"queryNamespaceMappings returned 1 errors\\"" agentID=cd482abd-7606-4364-a30b-3105066e76e7 component=agent-process type=postgres\_exporter > time="2025-08-25T15:29:07.124+00:00" level=error msg="ts=2025-08-25T15:29:07.124Z caller=server.go:130 level=error msg=\\"NAMESPACE ERRORS FOUND\\"" agentID=cd482abd-7606-4364-a30b-3105066e76e7 component=agent-process type=postgres\_exporter > time="2025-08-25T15:29:07.124+00:00" level=error msg="ts=2025-08-25T15:29:07.124Z caller=server.go:132 level=error namespace=pg\_custom\_replication\_wal msg=\\"Error running query on database \\\\\\"127.0.0.1:5432\\\\\\": pg\_custom\_replication\_wal pq: recovery is in progress\\"" agentID=cd482abd-7606-4364-a30b-3105066e76e7 component=agent-process type=postgres\_exporter > time="2025-08-25T15:29:07.124+00:00" level=error msg="ts=2025-08-25T15:29:07.124Z caller=postgres\_exporter.go:770 level=error err=\\"queryNamespaceMappings returned 1 errors\\"" agentID=cd482abd-7606-4364-a30b-3105066e76e7 component=agent-process type=postgres\_exporter > time="2025-08-25T15:29:07.125+00:00" level=info msg="ts=2025-08-25T15:29:07.125Z caller=log.go:245 msg=\\"handlererror gathering metrics: 11 error(s) occurred:\\\\n\* collected metric \\\\\\"pg\_custom\_database\_size\_custom\_bytes\\\\\\" { label:{name:\\\\\\"datname\\\\\\" value:\\\\\\"template1\\\\\\"} label:{name:\\\\\\"server\\\\\\" value:\\\\\\"127.0.0.1:5432\\\\\\"} gauge:{value:7.930383e+06}} was collected before with the same name and label values\\\\n\* collected metric \\\\\\"pg\_custom\_database\_size\_custom\_bytes\\\\\\" { label:{name:\\\\\\"datname\\\\\\" value:\\\\\\"template0\\\\\\"} label:{name:\\\\\\"server\\\\\\" value:\\\\\\"127.0.0.1:5432\\\\\\"} gauge:{value:7.725583e+06}} was collected before with the same name and label values\\\\n\* collected metric \\\\\\"pg\_custom\_database\_size\_custom\_bytes\\\\\\" { label:{name:\\\\\\"datname\\\\\\" value:\\\\\\"postgres\\\\\\"} label:{name:\\\\\\"server\\\\\\" value:\\\\\\"127.0.0.1:5432\\\\\\"} gauge:{value:8.113299e+06}} was collected before with the same name and label values\\\\n\* collected metric \\\\\\"pg\_custom\_database\_size\_custom\_bytes\\\\\\" { label:{name:\\\\\\"datname\\\\\\" value:\\\\\\"iamapp\\\\\\"} label:{name:\\\\\\"server\\\\\\" value:\\\\\\"127.0.0.1:5432\\\\\\"} gauge:{value:8.039571e+06}} was collected before with the same name and label values\\\\n\* collected metric \\\\\\"pg\_custom\_stat\_wal\_receiver\_written\_lsn\_bytes\\\\\\" { label:{name:\\\\\\"primary\_host\\\\\\" value:\\\\\\"iam-postgres-cluster-instance1-bn7t-0.iam-postgres-cluster-pods\\\\\\"} label:{name:\\\\\\"server\\\\\\" value:\\\\\\"127.0.0.1:5432\\\\\\"} label:{name:\\\\\\"status\\\\\\" value:\\\\\\"streaming\\\\\\"} gauge:{value:3.01989888e+08}} was collected before with the same name and label values\\\\n\* collected metric \\\\\\"pg\_custom\_stat\_wal\_receiver\_latest\_end\_lsn\_bytes\\\\\\" { label:{name:\\\\\\"primary\_host\\\\\\" value:\\\\\\"iam-postgres-cluster-instance1-bn7t-0.iam-postgres-cluster-pods\\\\\\"} label:{name:\\\\\\"server\\\\\\" value:\\\\\\"127.0.0.1:5432\\\\\\"} label:{name:\\\\\\"status\\\\\\" value:\\\\\\"streaming\\\\\\"} gauge:{value:3.01989888e+08}} was collected before with the same name and label values\\\\n\* collected metric \\\\\\"pg\_custom\_stat\_wal\_receiver\_lag\_bytes\\\\\\" { label:{name:\\\\\\"primary\_host\\\\\\" value:\\\\\\"iam-postgres-cluster-instance1-bn7t-0.iam-postgres-cluster-pods\\\\\\"} label:{name:\\\\\\"server\\\\\\" value:\\\\\\"127.0.0.1:5432\\\\\\"} label:{name:\\\\\\"status\\\\\\" value:\\\\\\"streaming\\\\\\"} gauge:{value:0}} was collected before with the same name and label values\\\\n\* collected metric \\\\\\"pg\_custom\_stat\_wal\_receiver\_latest\_end\_time\_seconds\\\\\\" { label:{name:\\\\\\"primary\_host\\\\\\" value:\\\\\\"iam-postgres-cluster-instance1-bn7t-0.iam-postgres-cluster-pods\\\\\\"} label:{name:\\\\\\"server\\\\\\" value:\\\\\\"127.0.0.1:5432\\\\\\"} label:{name:\\\\\\"status\\\\\\" value:\\\\\\"streaming\\\\\\"} gauge:{value:1.756135467307919e+09}} was collected before with the same name and label values\\\\n\* collected metric \\\\\\"pg\_custom\_stat\_wal\_receiver\_lag\_time\_seconds\\\\\\" { label:{name:\\\\\\"primary\_host\\\\\\" value:\\\\\\"iam-postgres-cluster-instance1-bn7t-0.iam-postgres-cluster-pods\\\\\\"} label:{name:\\\\\\"server\\\\\\" value:\\\\\\"127.0.0.1:5432\\\\\\"} label:{name:\\\\\\"status\\\\\\" value:\\\\\\"streaming\\\\\\"} gauge:{value:279.816133}} was collected before with the same name and label values\\\\n\* collected metric \\\\\\"pg\_custom\_stat\_wal\_receiver\_last\_msg\_send\_time\_seconds\\\\\\" { label:{name:\\\\\\"primary\_host\\\\\\" value:\\\\\\"iam-postgres-cluster-instance1-bn7t-0.iam-postgres-cluster-pods\\\\\\"} label:{name:\\\\\\"server\\\\\\" value:\\\\\\"127.0.0.1:5432\\\\\\"} label:{name:\\\\\\"status\\\\\\" value:\\\\\\"streaming\\\\\\"} gauge:{value:1.7561357382459e+09}} was collected before with the same name and label values\\\\n\* collected metric \\\\\\"pg\_custom\_stat\_wal\_receiver\_last\_msg\_receipt\_time\_seconds\\\\\\" { label:{name:\\\\\\"primary\_host\\\\\\" value:\\\\\\"iam-postgres-cluster-instance1-bn7t-0.iam-postgres-cluster-pods\\\\\\"} label:{name:\\\\\\"server\\\\\\" value:\\\\\\"127.0.0.1:5432\\\\\\"} label:{name:\\\\\\"status\\\\\\" value:\\\\\\"streaming\\\\\\"} gauge:{value:1.756135738258575e+09}} was collected before with the same name and label values\\"" agentID=cd482abd-7606-4364-a30b-3105066e76e7 component=agent-process type=postgres\_exporter > \`\`\` > Meanwhile, the sidecar pmm-client running next to the primary instance reports: > \`\`\` > time="2025-08-25T15:31:53.632+00:00" level=info msg="ts=2025-08-25T15:31:53.631Z caller=percona\_exporter.go:86 msg=\\"Excluded databases\\" databases=\\"\[template0 template1 cloudsqladmin pmm-managed-dev azure\_maintenance rdsadmin\]\\"" agentID=de1550a3-b421-4f31-a35e-9400e55448fe component=agent-process type=postgres\_exporter > time="2025-08-25T15:31:53.658+00:00" level=info msg="ts=2025-08-25T15:31:53.658Z caller=log.go:245 msg=\\"handlererror gathering metrics: 22 error(s) occurred:\\\\n\* collected metric \\\\\\"pg\_custom\_stat\_activity\_walsender\_pid\\\\\\" { label:{name:\\\\\\"application\_name\\\\\\" value:\\\\\\"iam-postgres-cluster-instance1-2lg8-0\\\\\\"} label:{name:\\\\\\"client\_addr\\\\\\" value:\\\\\\"10.42.5.19\\\\\\"} label:{name:\\\\\\"server\\\\\\" value:\\\\\\"127.0.0.1:5432\\\\\\"} label:{name:\\\\\\"state\\\\\\" value:\\\\\\"active\\\\\\"} label:{name:\\\\\\"usename\\\\\\" value:\\\\\\"\_crunchyrepl\\\\\\"} gauge:{value:752}} was collected before with the same name and label values\\\\n\* collected metric \\\\\\"pg\_custom\_stat\_activity\_walsender\_backend\_start\_unix\\\\\\" { label:{name:\\\\\\"application\_name\\\\\\" value:\\\\\\"iam-postgres-cluster-instance1-2lg8-0\\\\\\"} label:{name:\\\\\\"client\_addr\\\\\\" value:\\\\\\"10.42.5.19\\\\\\"} label:{name:\\\\\\"server\\\\\\" value:\\\\\\"127.0.0.1:5432\\\\\\"} label:{name:\\\\\\"state\\\\\\" value:\\\\\\"active\\\\\\"} label:{name:\\\\\\"usename\\\\\\" value:\\\\\\"\_crunchyrepl\\\\\\"} gauge:{value:1.756135133033277e+09}} was collected before with the same name and label values\\\\n\* collected metric \\\\\\"pg\_custom\_stat\_activity\_walsender\_pid\\\\\\" { label:{name:\\\\\\"application\_name\\\\\\" value:\\\\\\"iam-postgres-cluster-instance1-4l48-0\\\\\\"} label:{name:\\\\\\"client\_addr\\\\\\" value:\\\\\\"10.42.4.19\\\\\\"} label:{name:\\\\\\"server\\\\\\" value:\\\\\\"127.0.0.1:5432\\\\\\"} label:{name:\\\\\\"state\\\\\\" value:\\\\\\"active\\\\\\"} label:{name:\\\\\\"usename\\\\\\" value:\\\\\\"\_crunchyrepl\\\\\\"} gauge:{value:374}} was collected before with the same name and label values\\\\n\* collected metric \\\\\\"pg\_custom\_stat\_activity\_walsender\_backend\_start\_unix\\\\\\" { label:{name:\\\\\\"application\_name\\\\\\" value:\\\\\\"iam-postgres-cluster-instance1-4l48-0\\\\\\"} label:{name:\\\\\\"client\_addr\\\\\\" value:\\\\\\"10.42.4.19\\\\\\"} label:{name:\\\\\\"server\\\\\\" value:\\\\\\"127.0.0.1:5432\\\\\\"} label:{name:\\\\\\"state\\\\\\" value:\\\\\\"active\\\\\\"} label:{name:\\\\\\"usename\\\\\\" value:\\\\\\"\_crunchyrepl\\\\\\"} gauge:{value:1.756135109421014e+09}} was collected before with the same name and label values\\\\n\* collected metric \\\\\\"pg\_custom\_database\_size\_custom\_bytes\\\\\\" { label:{name:\\\\\\"datname\\\\\\" value:\\\\\\"template1\\\\\\"} label:{name:\\\\\\"server\\\\\\" value:\\\\\\"127.0.0.1:5432\\\\\\"} gauge:{value:8.088723e+06}} was collected before with the same name and label values\\\\n\* collected metric \\\\\\"pg\_custom\_database\_size\_custom\_bytes\\\\\\" { label:{name:\\\\\\"datname\\\\\\" value:\\\\\\"template0\\\\\\"} label:{name:\\\\\\"server\\\\\\" value:\\\\\\"127.0.0.1:5432\\\\\\"} gauge:{value:7.725583e+06}} was collected before with the same name and label values\\\\n\* collected metric \\\\\\"pg\_custom\_database\_size\_custom\_bytes\\\\\\" { label:{name:\\\\\\"datname\\\\\\" value:\\\\\\"postgres\\\\\\"} label:{name:\\\\\\"server\\\\\\" value:\\\\\\"127.0.0.1:5432\\\\\\"} gauge:{value:8.113299e+06}} was collected before with the same name and label values\\\\n\* collected metric \\\\\\"pg\_custom\_database\_size\_custom\_bytes\\\\\\" { label:{name:\\\\\\"datname\\\\\\" value:\\\\\\"iamapp\\\\\\"} label:{name:\\\\\\"server\\\\\\" value:\\\\\\"127.0.0.1:5432\\\\\\"} gauge:{value:8.039571e+06}} was collected before with the same name and label values\\\\n\* collected metric \\\\\\"pg\_custom\_stat\_replication\_flush\_lsn\_bytes\\\\\\" { label:{name:\\\\\\"application\_name\\\\\\" value:\\\\\\"iam-postgres-cluster-instance1-4l48-0\\\\\\"} label:{name:\\\\\\"client\_addr\\\\\\" value:\\\\\\"10.42.4.19\\\\\\"} label:{name:\\\\\\"pid\\\\\\" value:\\\\\\"374\\\\\\"} label:{name:\\\\\\"server\\\\\\" value:\\\\\\"127.0.0.1:5432\\\\\\"} label:{name:\\\\\\"state\\\\\\" value:\\\\\\"streaming\\\\\\"} label:{name:\\\\\\"sync\_state\\\\\\" value:\\\\\\"async\\\\\\"} label:{name:\\\\\\"usename\\\\\\" value:\\\\\\"\_crunchyrepl\\\\\\"} gauge:{value:3.01989888e+08}} was collected before with the same name and label values\\\\n\* collected metric \\\\\\"pg\_custom\_stat\_replication\_write\_lsn\_bytes\\\\\\" { label:{name:\\\\\\"application\_name\\\\\\" value:\\\\\\"iam-postgres-cluster-instance1-4l48-0\\\\\\"} label:{name:\\\\\\"client\_addr\\\\\\" value:\\\\\\"10.42.4.19\\\\\\"} label:{name:\\\\\\"pid\\\\\\" value:\\\\\\"374\\\\\\"} label:{name:\\\\\\"server\\\\\\" value:\\\\\\"127.0.0.1:5432\\\\\\"} label:{name:\\\\\\"state\\\\\\" value:\\\\\\"streaming\\\\\\"} label:{name:\\\\\\"sync\_state\\\\\\" value:\\\\\\"async\\\\\\"} label:{name:\\\\\\"usename\\\\\\" value:\\\\\\"\_crunchyrepl\\\\\\"} gauge:{value:3.01989888e+08}} was collected before with the same name and label values\\\\n\* collected metric \\\\\\"pg\_custom\_stat\_replication\_replay\_lsn\_bytes\\\\\\" { label:{name:\\\\\\"application\_name\\\\\\" value:\\\\\\"iam-postgres-cluster-instance1-4l48-0\\\\\\"} label:{name:\\\\\\"client\_addr\\\\\\" value:\\\\\\"10.42.4.19\\\\\\"} label:{name:\\\\\\"pid\\\\\\" value:\\\\\\"374\\\\\\"} label:{name:\\\\\\"server\\\\\\" value:\\\\\\"127.0.0.1:5432\\\\\\"} label:{name:\\\\\\"state\\\\\\" value:\\\\\\"streaming\\\\\\"} label:{name:\\\\\\"sync\_state\\\\\\" value:\\\\\\"async\\\\\\"} label:{name:\\\\\\"usename\\\\\\" value:\\\\\\"\_crunchyrepl\\\\\\"} gauge:{value:3.01989888e+08}} was collected before with the same name and label values\\\\n\* collected metric \\\\\\"pg\_custom\_stat\_replication\_write\_lag\_seconds\\\\\\" { label:{name:\\\\\\"application\_name\\\\\\" value:\\\\\\"iam-postgres-cluster-instance1-4l48-0\\\\\\"} label:{name:\\\\\\"client\_addr\\\\\\" value:\\\\\\"10.42.4.19\\\\\\"} label:{name:\\\\\\"pid\\\\\\" value:\\\\\\"374\\\\\\"} label:{name:\\\\\\"server\\\\\\" value:\\\\\\"127.0.0.1:5432\\\\\\"} label:{name:\\\\\\"state\\\\\\" value:\\\\\\"streaming\\\\\\"} label:{name:\\\\\\"sync\_state\\\\\\" value:\\\\\\"async\\\\\\"} label:{name:\\\\\\"usename\\\\\\" value:\\\\\\"\_crunchyrepl\\\\\\"} gauge:{value:nan}} was collected before with the same name and label values\\\\n\* collected metric \\\\\\"pg\_custom\_stat\_replication\_flush\_lag\_seconds\\\\\\" { label:{name:\\\\\\"application\_name\\\\\\" value:\\\\\\"iam-postgres-cluster-instance1-4l48-0\\\\\\"} label:{name:\\\\\\"client\_addr\\\\\\" value:\\\\\\"10.42.4.19\\\\\\"} label:{name:\\\\\\"pid\\\\\\" value:\\\\\\"374\\\\\\"} label:{name:\\\\\\"server\\\\\\" value:\\\\\\"127.0.0.1:5432\\\\\\"} label:{name:\\\\\\"state\\\\\\" value:\\\\\\"streaming\\\\\\"} label:{name:\\\\\\"sync\_state\\\\\\" value:\\\\\\"async\\\\\\"} label:{name:\\\\\\"usename\\\\\\" value:\\\\\\"\_crunchyrepl\\\\\\"} gauge:{value:nan}} was collected before with the same name and label values\\\\n\* collected metric \\\\\\"pg\_custom\_stat\_replication\_replay\_lag\_seconds\\\\\\" { label:{name:\\\\\\"application\_name\\\\\\" value:\\\\\\"iam-postgres-cluster-instance1-4l48-0\\\\\\"} label:{name:\\\\\\"client\_addr\\\\\\" value:\\\\\\"10.42.4.19\\\\\\"} label:{name:\\\\\\"pid\\\\\\" value:\\\\\\"374\\\\\\"} label:{name:\\\\\\"server\\\\\\" value:\\\\\\"127.0.0.1:5432\\\\\\"} label:{name:\\\\\\"state\\\\\\" value:\\\\\\"streaming\\\\\\"} label:{name:\\\\\\"sync\_state\\\\\\" value:\\\\\\"async\\\\\\"} label:{name:\\\\\\"usename\\\\\\" value:\\\\\\"\_crunchyrepl\\\\\\"} gauge:{value:nan}} was collected before with the same name and label values\\\\n\* collected metric \\\\\\"pg\_custom\_stat\_replication\_flush\_lsn\_bytes\\\\\\" { label:{name:\\\\\\"application\_name\\\\\\" value:\\\\\\"iam-postgres-cluster-instance1-2lg8-0\\\\\\"} label:{name:\\\\\\"client\_addr\\\\\\" value:\\\\\\"10.42.5.19\\\\\\"} label:{name:\\\\\\"pid\\\\\\" value:\\\\\\"752\\\\\\"} label:{name:\\\\\\"server\\\\\\" value:\\\\\\"127.0.0.1:5432\\\\\\"} label:{name:\\\\\\"state\\\\\\" value:\\\\\\"streaming\\\\\\"} label:{name:\\\\\\"sync\_state\\\\\\" value:\\\\\\"async\\\\\\"} label:{name:\\\\\\"usename\\\\\\" value:\\\\\\"\_crunchyrepl\\\\\\"} gauge:{value:3.01989888e+08}} was collected before with the same name and label values\\\\n\* collected metric \\\\\\"pg\_custom\_stat\_replication\_write\_lsn\_bytes\\\\\\" { label:{name:\\\\\\"application\_name\\\\\\" value:\\\\\\"iam-postgres-cluster-instance1-2lg8-0\\\\\\"} label:{name:\\\\\\"client\_addr\\\\\\" value:\\\\\\"10.42.5.19\\\\\\"} label:{name:\\\\\\"pid\\\\\\" value:\\\\\\"752\\\\\\"} label:{name:\\\\\\"server\\\\\\" value:\\\\\\"127.0.0.1:5432\\\\\\"} label:{name:\\\\\\"state\\\\\\" value:\\\\\\"streaming\\\\\\"} label:{name:\\\\\\"sync\_state\\\\\\" value:\\\\\\"async\\\\\\"} label:{name:\\\\\\"usename\\\\\\" value:\\\\\\"\_crunchyrepl\\\\\\"} gauge:{value:3.01989888e+08}} was collected before with the same name and label values\\\\n\* collected metric \\\\\\"pg\_custom\_stat\_replication\_replay\_lsn\_bytes\\\\\\" { label:{name:\\\\\\"application\_name\\\\\\" value:\\\\\\"iam-postgres-cluster-instance1-2lg8-0\\\\\\"} label:{name:\\\\\\"client\_addr\\\\\\" value:\\\\\\"10.42.5.19\\\\\\"} label:{name:\\\\\\"pid\\\\\\" value:\\\\\\"752\\\\\\"} label:{name:\\\\\\"server\\\\\\" value:\\\\\\"127.0.0.1:5432\\\\\\"} label:{name:\\\\\\"state\\\\\\" value:\\\\\\"streaming\\\\\\"} label:{name:\\\\\\"sync\_state\\\\\\" value:\\\\\\"async\\\\\\"} label:{name:\\\\\\"usename\\\\\\" value:\\\\\\"\_crunchyrepl\\\\\\"} gauge:{value:3.01989888e+08}} was collected before with the same name and label values\\\\n\* collected metric \\\\\\"pg\_custom\_stat\_replication\_write\_lag\_seconds\\\\\\" { label:{name:\\\\\\"application\_name\\\\\\" value:\\\\\\"iam-postgres-cluster-instance1-2lg8-0\\\\\\"} label:{name:\\\\\\"client\_addr\\\\\\" value:\\\\\\"10.42.5.19\\\\\\"} label:{name:\\\\\\"pid\\\\\\" value:\\\\\\"752\\\\\\"} label:{name:\\\\\\"server\\\\\\" value:\\\\\\"127.0.0.1:5432\\\\\\"} label:{name:\\\\\\"state\\\\\\" value:\\\\\\"streaming\\\\\\"} label:{name:\\\\\\"sync\_state\\\\\\" value:\\\\\\"async\\\\\\"} label:{name:\\\\\\"usename\\\\\\" value:\\\\\\"\_crunchyrepl\\\\\\"} gauge:{value:nan}} was collected before with the same name and label values\\\\n\* collected metric \\\\\\"pg\_custom\_stat\_replication\_flush\_lag\_seconds\\\\\\" { label:{name:\\\\\\"application\_name\\\\\\" value:\\\\\\"iam-postgres-cluster-instance1-2lg8-0\\\\\\"} label:{name:\\\\\\"client\_addr\\\\\\" value:\\\\\\"10.42.5.19\\\\\\"} label:{name:\\\\\\"pid\\\\\\" value:\\\\\\"752\\\\\\"} label:{name:\\\\\\"server\\\\\\" value:\\\\\\"127.0.0.1:5432\\\\\\"} label:{name:\\\\\\"state\\\\\\" value:\\\\\\"streaming\\\\\\"} label:{name:\\\\\\"sync\_state\\\\\\" value:\\\\\\"async\\\\\\"} label:{name:\\\\\\"usename\\\\\\" value:\\\\\\"\_crunchyrepl\\\\\\"} gauge:{value:nan}} was collected before with the same name and label values\\\\n\* collected metric \\\\\\"pg\_custom\_stat\_replication\_replay\_lag\_seconds\\\\\\" { label:{name:\\\\\\"application\_name\\\\\\" value:\\\\\\"iam-postgres-cluster-instance1-2lg8-0\\\\\\"} label:{name:\\\\\\"client\_addr\\\\\\" value:\\\\\\"10.42.5.19\\\\\\"} label:{name:\\\\\\"pid\\\\\\" value:\\\\\\"752\\\\\\"} label:{name:\\\\\\"server\\\\\\" value:\\\\\\"127.0.0.1:5432\\\\\\"} label:{name:\\\\\\"state\\\\\\" value:\\\\\\"streaming\\\\\\"} label:{name:\\\\\\"sync\_state\\\\\\" value:\\\\\\"async\\\\\\"} label:{name:\\\\\\"usename\\\\\\" value:\\\\\\"\_crunchyrepl\\\\\\"} gauge:{value:nan}} was collected before with the same name and label values\\\\n\* collected metric \\\\\\"pg\_custom\_replication\_wal\_received\_lsn\\\\\\" { label:{name:\\\\\\"server\\\\\\" value:\\\\\\"127.0.0.1:5432\\\\\\"} gauge:{value:nan}} was collected before with the same name and label values\\\\n\* collected metric \\\\\\"pg\_custom\_replication\_wal\_lag\_bytes\\\\\\" { label:{name:\\\\\\"server\\\\\\" value:\\\\\\"127.0.0.1:5432\\\\\\"} gauge:{value:5.0331648e+07}} was collected before with the same name and label values\\"" agentID=de1550a3-b421-4f31-a35e-9400e55448fe component=agent-process type=postgres\_exporter > \`\`\` > > As you can see, these log lines are quite long. And as they indicate, some queries are being run against an instance that is 'recovering', meaning it is in the state of ingesting changes from the main instance. > > > \### More about the problem > > Expected behavior would be that the pmm-client (or its postgres\_exporter) skips some queries the instance cannot answer because of its cluster position of being a replica. If that is not possible, it should not log such kind of failed queries as 'errors'. > > \### Steps to reproduce > > 1. Start a postgres cluster with the following: > \`\`\`yaml > apiVersion: pgv2.percona.com/v2 > kind: PerconaPGCluster > metadata: > name: cluster1 > annotations: > pgv2.percona.com/custom-patroni-version: "4" # this is set because patroni version can get issued on arm64 nodes in my cluster > spec: > crVersion: 2.7.0 > image: docker.io/percona/percona-postgresql-operator:2.7.0-ppg17.5.2-postgres > imagePullPolicy: Always > postgresVersion: 17 > instances: > - name: instance1 > replicas: 3 > affinity: > nodeAffinity: > requiredDuringSchedulingIgnoredDuringExecution: > nodeSelectorTerms: > - matchExpressions: > - key: mycompany.com/run > operator: In > values: \["db" \] > dataVolumeClaimSpec: > storageClassName: longhorn-fast > accessModes: > - ReadWriteOnce > resources: > requests: > storage: 1Gi > proxy: > pgBouncer: > replicas: 3 > image: docker.io/percona/percona-pgbouncer:1.24.1 > > backups: > pgbackrest: > image: docker.io/percona/percona-pgbackrest:2.55.0 > repoHost: {} > manual: > repoName: repo1 > options: > - --type=full > repos: > - name: repo1 > schedules: > full: "0 0 \* \* 6" > volume: > volumeClaimSpec: > storageClassName: longhorn-fast > accessModes: > - ReadWriteOnce > resources: > requests: > storage: 1Gi > pmm: > enabled: true > image: docker.io/percona/pmm-client:3.3.1 > secret: pmm-iam-service-account-secret > serverHost: monitoring-service.pmm.svc.cluster.local > \`\`\` > 2. Wait for the pmm client on a replica instance to connect to its db. > 3. Observe something like: > \`\`\` > time="2025-08-26T10:58:20.201+00:00" level=error msg="ts=2025-08-26T10:58:20.201Z caller=namespace.go:236 level=error err=\\"Error running query on database \\\\\\"127.0.0.1:5432\\\\\\": pg\_custom\_replication\_wal pq: recovery is in progress\\"" agentID=b8712013-dec7-40ec-8ba0-0a88e92bb969 component=agent-process type=postgres\_exporter > time="2025-08-26T10:58:20.214+00:00" level=error msg="ts=2025-08-26T10:58:20.214Z caller=server.go:130 level=error msg=\\"NAMESPACE ERRORS FOUND\\"" agentID=b8712013-dec7-40ec-8ba0-0a88e92bb969 component=agent-process type=postgres\_exporter > time="2025-08-26T10:58:20.214+00:00" level=error msg="ts=2025-08-26T10:58:20.214Z caller=server.go:132 level=error namespace=pg\_custom\_replication\_wal msg=\\"Error running query on database \\\\\\"127.0.0.1:5432\\\\\\": pg\_custom\_replication\_wal pq: recovery is in progress\\"" agentID=b8712013-dec7-40ec-8ba0-0a88e92bb969 component=agent-process type=postgres\_exporter > time="2025-08-26T10:58:20.214+00:00" level=error msg="ts=2025-08-26T10:58:20.214Z caller=postgres\_exporter.go:770 level=error err=\\"queryNamespaceMappings returned 1 errors\\"" agentID=b8712013-dec7-40ec-8ba0-0a88e92bb969 component=agent-process type=postgres\_exporter > time="2025-08-26T10:58:23.310+00:00" level=info msg="ts=2025-08-26T10:58:23.310Z caller=percona\_exporter.go:86 msg=\\"Excluded databases\\" databases=\\"\[template0 template1 cloudsqladmin pmm-managed-dev azure\_maintenance rdsadmin\]\\"" agentID=b8712013-dec7-40ec-8ba0-0a88e92bb969 component=agent-process type=postgres\_exporter > > \`\`\` > > \### Versions > > 1. Kubernetes: v1.33.3+k3s1 > 2. Operator: \[v2.7.0\](https://raw.githubusercontent.com/percona/percona-postgresql-operator/refs/tags/v2.7.0/deploy/bundle.yaml) > 3. Database: tested with both 2.7.0-ppg16.9-postgres and 2.7.0-ppg17.5.2-postgres > > > > \### Anything else? > > There are some related things I found: > \- https://forums.percona.com/t/pmm3-postgres-on-read-only-pg-custom-replication-wal-pq-recovery-is-in-progress/38681 > \- https://github.com/prometheus-community/postgres\_exporter/issues/962 > > I use Selinux on my nodes, maybe that could cause issues?