Pmm-client sidecar container is not getting created

Description:

We have been creating pmm-server and pmm-client side car containers but unable to deploy pmm-client side car containers on primary Patroni and standby Patroni cluster on different OCP environments

Steps to Reproduce:

[root@bastion-node pg-db]# oc get pods -n pmm
NAME READY STATUS RESTARTS AGE
pmm-server-f56bd79c9-g2s8x 1/1 Running 0 4d9h
[root@bastion-node pg-db]#

[root@bastion-node pg-db]# cat values.yaml

Default values for pg-cluster.

This is a YAML-formatted file.

Declare variables to be passed into your templates.

annotations:

test-annotation: value

finalizers:

Set this if you want that operator deletes the PVCs on cluster deletion

- percona.com/delete-pvc

Set this if you want that operator deletes the ssl objects on cluster deletion

- percona.com/delete-ssl

Set this if you want that operator deletes the backups on cluster deletion

- percona.com/delete-backups

crVersion: 2.8.2

example-annotation: value

labels:

example-label: value

service:

Valid type are NodePort or LoadBalancer. Defaults to NodePort

type: NodePort

If Postgrescluster has to be deployed on the openshift env, set openshift: true.

openshift: true

repository: registry.connect.redhat.com/percona/percona-postgresql-operator-containers
image: registry.connect.redhat.com/percona/percona-postgresql-operator-containers:2.8.2-ppg-16-postgres
imagePullPolicy: Always
imagePullSecrets:

  • name: rh-catalog
    postgresVersion: 16

port: 5432

pause: false
unmanaged: false
standby:
enabled: false

host: “”

port: “”

repoName: repo1

customRootCATLSSecret:

name: cluster1-ca-cert

items:

- key: “tls.crt”

path: “root.crt”

- key: “tls.key”

path: “root.key”

customTLSSecret:
name: “”
customReplicationTLSSecret:
name: “”

volumes:

instances:

  • name: pgsa
    replicas: 2

    expose primary cluster IP for streaming replication

expose:

type: ClusterIP

dataVolumeClaimSpec:
storageClassName: ocs-storagecluster-ceph-rbd
accessModes:

  • ReadWriteOnce
    resources:
    requests:
    storage: 5Gi

affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:

  • matchExpressions:
  • key: database
    operator: In # The “equals” logic
    values:
  • postgres # The label value on your node
podAntiAffinity:
  requiredDuringSchedulingIgnoredDuringExecution:
  - labelSelector:
      matchLabels:
        postgres-operator.crunchydata.com/data: postgres
    topologyKey: kubernetes.io/hostname

tablespaceVolumes:

- name: user

dataVolumeClaimSpec:

accessModes:

- ‘ReadWriteOnce’

resources:

requests:

storage: 1Gi

proxy:
pgBouncer:
replicas: 0
image: docker.io/percona/percona-pgbouncer:1.25.0-1

env:

- name: MY_ENV

value: “1000”

envFrom:

- secretRef:

name: pgbouncer-env-secret

exposeSuperusers: true

resources:

limits:

cpu: 200m

memory: 128Mi

containers:

pgbouncerConfig:

resources:

limits:

cpu: 200m

memory: 128Mi

expose:

annotations:

my-annotation: value1

labels:

my-label: value2

type: LoadBalancer

loadBalancerSourceRanges:

- 10.0.0.0/8

affinity:
  podAntiAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
    - weight: 1
      podAffinityTerm:
        labelSelector:
          matchLabels:
            postgres-operator.crunchydata.com/role: pgbouncer
        topologyKey: kubernetes.io/hostname

backups:
enabled: true
trackLatestRestorableTime: true

pgbackrest:
image: registry.connect.redhat.com/percona/percona-postgresql-operator-containers:2.8.2-ppg-16-pgbackrest
env:
envFrom:

repos:
  - name: repo1
    schedules:
      full: "12 1 * * *"
      differential: "01 11 * * 1-6"
    volume:
      volumeClaimSpec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 5Gi
global:
  repo1-retention-full: "2"
  repo1-retention-diff: "7"
  repo1-retention-full-type: time
  repo1-retention-diff-type: time
  repo1-cipher-type: none

repoHost:
  affinity:
    podAntiAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
        - weight: 1
          podAffinityTerm:
            labelSelector:
              matchLabels:
                postgres-operator.crunchydata.com/data: pgbackrest
            topologyKey: kubernetes.io/hostname

backupaffinity:
enabled: true
nodeSelectorTerms:
key: database
value: postgres

pmm:
enabled: true
image:
repository: registry.connect.redhat.com/percona/percona-postgresql-operator-containers
tag: 2.3.1-ppg-16-pmm-client

imagePullPolicy: IfNotPresent

secret: cluster1-pmm-secret

serverHost: “pmm-server.percona.svc.cluster.local”
serverSecret: pmm-server-secret
resources:
requests:
memory: 256Mi
cpu: 500m

secrets:
name:

replication user password

primaryuser:

superuser password

postgres: postgres

pgbouncer user password

pgbouncer:

pguser user password

pguser:
[root@bastion-node pg-db]#

Version:

Operator 2.8 with Postgres 16 and PMM 3

Logs:

No errors in operator logs and there were no errors in pod describe output

Expected Result:

Deploy PMM clients on Primary and Standby Patroni clusters

Actual Result:

Unable to deploy PMM client

Additional Information: Standby values.yaml configuration

[root@idso-ocp3-bastion pg-db]# cat standby-values.yaml

Values.yaml is common for setting up Primary and Standby database.

Select what to install

– Install Percona Standby Postgresql Cluster. Set it to true if you want to install standby DB ( this requires Primary postgresql Daabase is up and running )

Enter the namespace meant for couchbase cluster deployment. if not specified, it defaults .Release.Namespace

namespace: percona

If Crunchy PostgreSQL is being deployed in IPv6-only k8s environment then set IPv6: true

IPv6:

– If install.couchbaseBackup CouchbaseBackups runs a job which preserves data into backups

backups:
enabled: true

pgbackrest:

imagePgBackRest is the pgBackRest backup utility image.

For RHOS cluster, download images from offical redhat contianer register using https://catalog.redhat.com/search?gs&q=couchbase&searchType=containers

And push it to customer image respository and use obsolete path

image: registry.connect.redhat.com/percona/percona-postgresql-operator-containers:2.8.2-ppg-16-pgbackrest

repoHost:
  tolerations:
    - key: "node.ocs.openshift.io/storage"
      operator: "Equal"
      value: "true"
      effect: "NoSchedule"

repos:
  - name: repo1

    schedules:
      full: "12 1 * * *"
      differential: "01 11 * * 1-6"

    volume:
      volumeClaimSpec:
        storageClassName: openebs-hostpath

        accessModes:
          - ReadWriteOnce

        resources:
          requests:
            storage: 5Gi

Enter CPU & RAM resevered for the backup Pod job. If not specified, it defaults to 4 CPUs and 8Gi RAM.

job:
cpu: 250m
memory: 250Mi

If baackup has to run on a specific worker nodes, then change the flag enabled: true and replace : with actual label to be used

Below section assumes that worker nodes are labeled.

backupaffinity:
enabled: true
nodeSelectorTerms:
key: database
value: postgres

If Postgrescluster has to be deployed on the openshift env, set openshift: true.

openshift: true

imagePostgres is a Postgres image.

For RHOS cluster, download images from offical redhat contianer register using https://catalog.redhat.com/search?gs&q=couchbase&searchType=containers

And push it to customer image respository and use obsolete path

imagePostgres: registry.connect.redhat.com/percona/percona-postgresql-operator-containers:2.8.2-ppg-16-postgres

imagePullSecrets references Secrets that credentials for pulling image from private repositories

imagePullSecrets:

  • name: rh-catalog

postgresVersion sets the version to deploy. This value is required.

Obtain the value “.Values.imagePostgres”

Valid values are “14” or “15” or “16”

postgresVersion: 16

Below is the standby database section.

standby:
enabled: true
host: 192.168.101.15
port: 30007

customTLSSecret:
name: pg-cluster-secondary-cluster-cert

customReplicationTLSSecret:
name: pg-cluster-secondary-replication-cert

customRootCATLSSecret:
name: pg-cluster-secondary-ca-cert

instances:

  • name: dr1
    replicas: 1

    nodeSelector:
    kubernetes.io/hostname: idso-ocp3-cb-1.idso.lan

    tolerations:

    hostAliases:

    • ip: “192.168.101.15”
      hostnames:
      • “pg-cluster-ha-pg-db-primary.percona.svc.cluster.local”

    cpu: 2
    memory: 1Gi

    dataVolumeClaimSpec:
    storageClassName: openebs-hostpath
    accessModes:

    • ReadWriteOnce
      resources:
      requests:
      storage: 6Gi

    walVolumeClaimSpec:
    storageClassName: openebs-hostpath
    accessModes:

    • ReadWriteOnce
      resources:
      requests:
      storage: 6Gi

    affinity:
    podAntiAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:

shutdowns the entire cluster when set to true. By default, this is not set

shutdown:

patroni:
postgresql:

To retrieve details for the following parameters, follow these steps:

Visit https://pgtune.leopard.in.ua/

Enter DB version based on the .Values.postgresVersion

OStype:Linux

select “Online transaction processing system” as the DB type,

input CPU and RAM information based on the .Values.instances.cpu and .Values.instances.memory,

set the number of connections to 100,

lastly specify data storage type and click generate button.

max_connections: 100
shared_buffers: 256MB
effective_cache_size: 768MB
maintenance_work_mem: 64MB
checkpoint_completion_target: 0.9
wal_buffers: 7864kB
default_statistics_target: 100
work_mem: 1310kB
min_wal_size: 2GB
max_wal_size: 8GB
max_worker_processes: 2
max_parallel_workers_per_gather: 2

for optimimal performance, it is recommended to enable huge pages (huge_page: on). This setting depends on vm.nr_hugepages configured on worker nodes.

If vm.nr_hugepages isn’t set on worker nodes or if Crunchy pods are deployed on nodes without vm.nr_hugepages configured, the pods will continuously terminate.

For PostgreSQL database, the required number of huge pages can be calculated using the formula: vm.nr_hugepages = (shared_buffers_in_MB / 2) + 100.

Valid values are on or off.

Valid values are on or off or try. Defaults to on.

huge_page: “off”

monitoring:
enabled: true

pmm:
enabled: true
image:
repository: docker.io/percona/pmm-client
tag: 3.4.1
serverHost: pmm-server-pmm.apps.lab.idso.lan
secret: pmm-secret
serverInsecureTLS: true

[root@idso-ocp3-bastion pg-db]#

Hi @Srinivas ,

I suggest you to use proper code blocks for terminal outputs, otherwise formatting makes it very hard to read.

Also, what errors in particular are you getting from the pmm-client sidecars, or from get pods outputs?

No errors in operator logs and there were no errors in pod describe output

Did you check the pmm-client sidecar container logs? Please also share the pod describe outputs, if you can.

Hi @Agustin_G

Thank you for your response.

The issue is fixed; We used the “username and password” in the PMM client to reference the PMM server, rather than the Service account Token when creating the secret for the PMM3 configuration.