Pmm-client is not starting up

PostgreSQL Percona Operator for PostgreSQL
1

Description:

I would like to monitor the postgresql cluster with pmm, but the client is not starting. I am using a bare-bones k8s cluster administered with Rancher.

Steps to Reproduce:

I followed the steps explained here:

I installed the pmm server in a separate namespace pmm and the service is coming up fine:

$ kubectl -n pmm get svc
NAME                 TYPE       CLUSTER-IP     EXTERNAL-IP   PORT(S)                      AGE
monitoring-service   NodePort   10.43.226.58   <none>        443:30975/TCP,80:31827/TCP   47h

I create an api key in the ui frontend (no expiry date) and base64-encode the string. I create a secret with the base64-encoded string:

apiVersion: v1
kind: Secret
metadata:
  name: cluster1-pmm-secret
type: Opaque
stringData:
  PMM_SERVER_KEY: "base64_encoded_string"

I apply the secret to the namespace where the PostgreSQL Operator and DB is running:

$ kubectl -n pgsql apply -f secret.yaml
$ kubectl -n pgsql get secret | grep cluster1
cluster1-pmm-secret                               Opaque                                1      47h
$ kubectl -n pgsql get svc | grep pg
pgsql-db-pg-db-ha                        ClusterIP   10.43.204.246   <none>        5432/TCP             37d
pgsql-db-pg-db-ha-config                 ClusterIP   None            <none>        <none>               37d
pgsql-db-pg-db-pgbouncer                 ClusterIP   10.43.29.204    <none>        5432/TCP             37d
pgsql-db-pg-db-pods                      ClusterIP   None            <none>        <none>               37d
pgsql-db-pg-db-primary                   ClusterIP   None            <none>        5432/TCP             37d
pgsql-db-pg-db-replicas                  ClusterIP   10.43.58.9      <none>        5432/TCP             37d

I then change the deploy/cr.yaml of pgsql-db and add the following:

pmm:
  enabled: true
  image:
    repository: percona/pmm-client
    tag: 2.44.0
  secret: cluster1-pmm-secret
  serverHost: monitoring-service.pmm.svc.cluster.local

I deploy the application. Nothing happens:

$ kubectl -n pgsql get pod | grep pg
pgsql-db-pg-db-instance1-zsjw-0                           4/4     Running     0              46h
pgsql-db-pg-db-pgbouncer-557765ffdf-6rtgv                 2/2     Running     2 (34d ago)    37d
pgsql-db-pg-db-repo-host-0                                2/2     Running     2 (34d ago)    37d
pgsql-db-pg-db-repo1-full-28988520-gkzhs                  0/1     Completed   0              29h
pgsql-pg-operator-7d55db47c-9449v                         1/1     Running     0              47h
$ kubectl -n pgsql logs -f pgsql-db-pg-db-instance1-zsjw-0 -c 
database               pgbackrest             postgres-startup       
nss-wrapper-init       pgbackrest-config      replication-cert-copy

Version:

PMM Server: v2.44.0, Helm Chart v1.3.21
PostgreSQL Operator: v2.3.1
PostgreSQL DB: v2.3.6
Kubernetes: v1.24.17
Rancher: v2.7.10

Expected Result:

A new pod for pmm-client (or a separate container within pgsql-db-pg-db-instance1-zsjw-0) should start in the namespace pgsql.

Actual Result:

Nothing happens, no client is started, pmm-server cannot monitor the db.