How to connect PMM to the quickstart postgres database

PostgreSQL
1

I created a database using the kubernetes quickstart here: With kubectl - Percona Operator for PostgreSQL

I can connect to it using:
kubectl run -i --rm --tty pg-client --image=perconalab/percona-distribution-postgresql:16 --restart=Never – psql $PGBOUNCER_URI

where the PGBOUNCER_URI is described in the following section.

So far, so good.

I then create a PMM instance using the instructions here: 5 Monitor the database with PMM - Percona Operator for PostgreSQL and I am able to connect with my browser and log in.

Also, so far, so good.

However, when I log in, the only database it seems to be monitoring is pmm-server-postgresql which I think belongs to the monitor itself, not the cluster1 database created in the first steps of the quickstart.

I’ve tried adding the cluster1 as a service with the hostname cluster1-pgbouncer.percona-postgres-operator.svc , however it insists on TLS certificates (can’t leave it blank and rely on defaults) and I have no idea where to get those from.

Also, the cluster1 user created in the quickstart does not have CREATEROLE privileges, so while I’m supposed to create a pmm user from what I read, I can’t do that either.

Suggestions and recommendations welcome.

2

Hi,

There is no need to manually add cluster to pmm server or create a user in pg if it is enabled in the cr. Could you check & share operator logs to see if there are any related errors in it? pls also check pmm-client container logs for any errors.

3

Hi, Eleanora, thanks for your reply.

The cr.yaml is the same one as given on your website for deploying the client with enabled changed to true.

For kubectl -n percona-postgres-operator logs deploy/percona-postgresql-operator , all of the log messages are at INFO, no errors. There is something about the token missing.

I’ve put the entire operator log in this gist: gist:d62ab156a140c67dbfa53ec40accb080 · GitHub .

% kubectl get pods -n percona-postgres-operator
NAME READY STATUS RESTARTS AGE
cluster1-backup-k5zc-pv4pc 0/1 Completed 0 44h
cluster1-instance1-4wpg-0 0/4 Init:ImagePullBackOff 0 43h
cluster1-instance1-kgl8-0 4/4 Running 0 44h
cluster1-instance1-ks4w-0 4/4 Running 0 44h
cluster1-pgbouncer-775cd78f4d-7g5lt 2/2 Running 0 19h
cluster1-pgbouncer-775cd78f4d-dkmlm 2/2 Running 0 43h
cluster1-pgbouncer-775cd78f4d-sgp6h 2/2 Running 0 43h
cluster1-repo-host-0 2/2 Running 0 19h
percona-postgresql-operator-6b887b756d-g442q 1/1 Running 1 (7h51m ago) 44h

For all of those, following the instructions on the Monitor the database with PMM page, I get “error: container pmm-client is not valid for pod” when I ask for the logs with -c pmm-client.

4

There is also this pod, somehow under the default namespace, but same error:

% kubectl get pods -n default
NAME                                                  READY   STATUS      RESTARTS   AGE
node-debugger-aks-nodes01-24440178-vmss000001-f6bpb   0/1     Completed   0          17h
pmm-0                                                 1/1     Running     0          43h

% kubectl logs -n default pmm-0 -c pmm-client
error: container pmm-client is not valid for pod pmm-0
5

As a further update, I created a token, updated the pmm client secrets using kubectl apply and then re-applied the cr to force a restart.

I even tried deleting the pods, but now they won’t come up:

% kubectl get pods -n percona-postgres-operator
NAME                                           READY   STATUS                  RESTARTS     AGE
cluster1-backup-k5zc-pv4pc                     0/1     Completed               0            46h
cluster1-instance1-4wpg-0                      0/5     Init:ImagePullBackOff   0            2m12s
cluster1-instance1-kgl8-0                      0/5     Init:ErrImagePull       0            2m10s
cluster1-instance1-ks4w-0                      0/5     Init:ImagePullBackOff   0            2m11s

% kubectl -n percona-postgres-operator logs cluster1-instance1-4wpg-0
Error from server (BadRequest): container “database” in pod “cluster1-instance1-4wpg-0” is waiting to start: PodInitializing
6

Hi,

To enable pmm client (make it run), the pmm should be enabled in cr and the token should be present in the secret. Until secret data is present - the pmm-client containers will not be created in pods and thus there will not be any stats in PMM server. As there was no token in secret - pmm-client containers were not created and the token error was in log.

In general there is no need to restart the cluster after the token is added - it should be restarted by operator itself (though may take some time to get triggered).

Although it looks strange that after reapplying the cr you’ve got ErrImagePull . If you still have this cluster - could you provide describe of one of the pods with ErrImagePull to check it?

Note: depending on PMM version you are running the TOKEN or KEY should be used (5 Monitor the database with PMM - Percona Operator for PostgreSQL)

There is also this pod, somehow under the default namespace, but same error:

This seems to be PMM server POD.

Please also recheck that spec.pmm.serverHost contains the host of the PMM server.

7

Hi,

Here’s the describe. I’m off to commute to work, will be online again in a couple of hours.

I specified a token, I believe I am using PMM 3.

8

So, we solved the pod restarting problem. Somehow, in the bundle.yaml,

image: docker.io/percona/percona-postgresql-operator:2.8.2 got set to 2.9 and since there is no 2.9, the docker image pull failed.

But now they are crash looping.

% kubectl get po -n 2percona
NAME READY STATUS RESTARTS AGE
cluster1-backup-n7m4-pc7fx 0/1 Completed 0 22h
cluster1-instance1-2lmb-0 4/5 CrashLoopBackOff 349 (4m59s ago) 22h
cluster1-instance1-8xbb-0 4/5 CrashLoopBackOff 351 (57s ago) 22h
cluster1-instance1-mvj8-0 4/5 CrashLoopBackOff 349 (3m47s ago) 22h
cluster1-pgbouncer-fd5d99bcc-6g8f7 2/2 Running 0 22h
cluster1-pgbouncer-fd5d99bcc-pfclc 2/2 Running 0 22h
cluster1-pgbouncer-fd5d99bcc-shq5g 2/2 Running 0 22h
cluster1-repo-host-0 2/2 Running 0 22h
percona-postgresql-operator-6b887b756d-lzqn4 1/1 Running 0 24h

I see the following containers:
% kubectl describe po -n 2percona cluster1-instance1-8xbb-0| fgrep -B1 containerd | fgrep -v containerd | sort
database-init:
database:
nss-wrapper-init:
pgbackrest-config:
pgbackrest:
pmm-client:
postgres-startup:
replication-cert-copy:

On the leader, 2lmb, I see (after removing the leader messages) in the database log:
2026-02-24 00:30:21,734 ERROR: ObjectCache.run ProtocolError(“Connection broken: ConnectionResetError(104, ‘Connection reset by peer’)”, ConnectionResetError(104, ‘Connection reset by peer’))
2026-02-24 05:03:34,674 ERROR: ObjectCache.run ProtocolError(“Connection broken: InvalidChunkLength(got length b’', 0 bytes read)”, InvalidChunkLength(got length b’‘, 0 bytes read))
2026-02-24 05:04:11,992 ERROR: ObjectCache.run ProtocolError("Connection broken: InvalidChunkLength(got length b’‘, 0 bytes read)", InvalidChunkLength(got length b’', 0 bytes read))

In the replication-cert-copy log, I see:
error: container replicationh-cert-copy is not valid for pod cluster1-instance1-2lmb-0

In the pmm-agent log, I see:
Checking local pmm-agent status…
pmm-agent is not running.
Config file /usr/local/percona/pmm/config/pmm-agent.yaml is not writable: no such file or directory.
time=“2026-02-24T18:10:49.950+00:00” level=info msg=“‘pmm-agent setup’ exited with 1” component=entrypoint
time=“2026-02-24T18:10:49.950+00:00” level=info msg=“Restarting pmm-agent setup in 5 seconds because PMM_AGENT_SIDECAR is enabled…” component=entrypoint
time=“2026-02-24T18:10:54.951+00:00” level=info msg=“Starting ‘pmm-agent setup’…” component=entrypoint
time=“2026-02-24T18:10:54.965+00:00” level=info msg=“Loading configuration file /usr/local/percona/pmm/config/pmm-agent.yaml.” component=setup
time=“2026-02-24T18:10:54.965+00:00” level=info msg=“Using /usr/local/percona/pmm2/exporters/node_exporter” component=setup
time=“2026-02-24T18:10:54.965+00:00” level=info msg=“Using /usr/local/percona/pmm2/exporters/mysqld_exporter” component=setup
time=“2026-02-24T18:10:54.965+00:00” level=info msg=“Using /usr/local/percona/pmm2/exporters/mongodb_exporter” component=setup
time=“2026-02-24T18:10:54.965+00:00” level=info msg=“Using /usr/local/percona/pmm2/exporters/postgres_exporter” component=setup
time=“2026-02-24T18:10:54.965+00:00” level=info msg=“Using /usr/local/percona/pmm2/exporters/proxysql_exporter” component=setup
time=“2026-02-24T18:10:54.965+00:00” level=info msg=“Using /usr/local/percona/pmm2/exporters/rds_exporter” component=setup
time=“2026-02-24T18:10:54.965+00:00” level=info msg=“Using /usr/local/percona/pmm2/exporters/azure_exporter” component=setup
time=“2026-02-24T18:10:54.965+00:00” level=info msg=“Using /usr/local/percona/pmm2/exporters/vmagent” component=setup
time=“2026-02-24T18:10:54.965+00:00” level=info msg=“Updating PMM Server address from "10.20.3.20" to "10.20.3.20:443".” component=setup
Checking local pmm-agent status…

And of course, nothing shows up in the PMM admin page.

9

spec.pmm.serverHost is correct.

10

Hi @Glenn_Glazer,

I reproduced this on k3d with PG Operator 2.8.2 and saw the same config-path error. Building on @eleonora.zinchenko’s guidance, the pmm-agent failure in post #8 lines up with the PMM client image your pods are running. Your gist shows perconalab/pmm-client:dev-latest, a development/CI image that only has the PMM2 directory structure (/usr/local/percona/pmm2/). The operator, running in PMM3 mode because your secret uses PMM_SERVER_TOKEN, sets the config path to /usr/local/percona/pmm/config/pmm-agent.yaml, which doesn’t exist in that image. Your pmm-agent logs in post #8 confirm this: the agent loads exporters from the PMM2 path (/usr/local/percona/pmm2/exporters/) but fails on the PMM3 config path.

In my test, the production image (percona/pmm-client:3.5.0) has the PMM3 directory and the sidecar starts correctly, while the dev image produces the exact error from your logs.

Fix: update spec.pmm.image in your CR to the certified image for operator 2.8.2:

spec:
  pmm:
    enabled: true
    image: percona/pmm-client:3.5.0
    serverHost: <your-pmm-server-host>
    secret: cluster1-pmm-secret

Your secret is already correct (using PMM_SERVER_TOKEN for PMM3). After applying the updated CR, the operator will roll the pods with the new image.

I did not reproduce the ConnectionResetError or replication-cert-copy errors from post #8 in my test. If they persist after the image change, can you share kubectl logs <pod> -c pmm-client and kubectl logs <pod> -c replication-cert-copy from a freshly restarted pod?

1 Like