PostgreSQL operator pods fail to come up

kd role/pgo-target-role -n percona-pgo
Name: pgo-target-role
Labels: vendor=crunchydata
Annotations:
PolicyRule:
Resources Non-Resource URLs Resource Names Verbs


configmaps [get list watch create patch update delete deletecollection]
endpoints [get list watch create patch update delete deletecollection]
persistentvolumeclaims [get list watch create patch update delete deletecollection]
pods/exec [get list watch create patch update delete deletecollection]
pods [get list watch create patch update delete deletecollection]
secrets [get list watch create patch update delete deletecollection]
services [get list watch create patch update delete deletecollection]
deployments.apps [get list watch create patch update delete deletecollection]
replicasets.apps [get list watch create patch update delete deletecollection]
jobs.batch [get list watch create patch update delete deletecollection]
pgclusters.crunchydata.com [get list watch create patch update delete deletecollection]
pgpolicies.crunchydata.com [get list watch create patch update delete deletecollection]
pgreplicas.crunchydata.com [get list watch create patch update delete deletecollection]
pgtasks.crunchydata.com [get list watch create patch update delete deletecollection]
pods/log [get list watch]

pgo-deploy-logs.txt (13.5 KB)
@Sergey_Pronin : Please find the attached logs from the pgo-deploy pod.

@Sergey_Pronin : I am looking for your help in resolving this issue.

Hey Ravi,

I will ask someone from the team to look into it. Seems you are hitting issues that are specific to your k8s deployment and it is not something that can be easily solved through a forum.

1 Like

@Sergey_Pronin : Thank you for following up with your team, I appreciate it.

Hey @Ilford ,

actually you can install the operator through a helm chart.
See this doc: Install with Helm - Percona Operator for PostgreSQL

You just need to set a version.

Hello @Ravi_Kumar_Pokala , were you able to solve the issue? I am facing similar error when deploying the operator

No, looks like operators need some changes to work.

1 Like

@Roua_Saad can you tell me more about your environment? Kubernetes version, the way you deploy the cluster, steps to reproduce the issue?

Sure! I was deploying Percona operator of version 1.2.0 in Openshift I followed this link Install on OpenShift - Percona Operator for PostgreSQL for the deployment and I used the same yaml files (for operator.yaml and cr.yaml) used with some modifications (namespace, images point it out to the image streams in the project);
the operator job was initialized (pgo-deploy) with its associated deployment pod ( with the 4 containers : apiserver , operator scheduler and event).
However when I deployed the cr.yaml file for Percona Distribution for PostgreSQL the Pgcluster and Pgreplica along with pgbouncer and so on were not created (no pods were initialized, expected to have as deployments postgres-cs-backrest-shared-repo, postgres-cs-pgbouncer, postgres-cs-repl1, postgres-cs-repl2 and postgres-cs).

So I looked into the operator container logs and I saw the following error:

oc apply -f operator.yaml –as=system:admin

Kindly find the operator yaml file:
apiVersion: v1
kind: ServiceAccount
metadata:
name: pgo-deployer-sa

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: pgo-deployer-cr
rules:

  • apiGroups:
    • ‘’
      resources:
    • namespaces
      verbs:
    • get
    • list
    • watch
    • create
    • update
    • delete
  • apiGroups:
    • ‘’
      resources:
    • serviceaccounts
      verbs:
    • get
    • create
    • update
    • delete
  • apiGroups:
    - rbac.authorization.k8s.io
    resources:
    • roles
    • rolebindings
      verbs:
    • get
    • create
    • update
    • delete
  • apiGroups:
    • ‘’
      resources:
    • configmaps
    • endpoints
    • pods
    • pods/exec
    • secrets
    • services
    • persistentvolumeclaims
      verbs:
    • get
    • list
    • watch
    • create
    • patch
    • update
    • delete
    • deletecollection
  • apiGroups:
    • ‘’
      resources:
      • pods/log
        verbs:
      • get
      • list
      • watch
  • apiGroups:
    • apps
      resources:
    • deployments
    • replicasets
      verbs:
    • get
    • list
    • watch
    • create
    • patch
    • update
    • delete
    • deletecollection
  • apiGroups:
    • batch
      resources:
    • jobs
      verbs:
    • get
    • list
    • watch
    • create
    • patch
    • update
    • delete
    • deletecollection
  • apiGroups:
    • pg.percona.com
      resources:
    • perconapgclusters
    • pgclusters
    • pgpolicies
    • pgreplicas
    • pgtasks
      verbs:
    • get
    • list
    • watch
    • create
    • patch
    • update
    • delete
    • deletecollection

apiVersion: v1
kind: ConfigMap
metadata:
name: pgo-deployer-cm
data:
values.yaml: |-
archive_mode: “true”
archive_timeout: “60”
ccp_image_pull_secret: “”
ccp_image_pull_secret_manifest: “”
create_rbac: “true”
delete_operator_namespace: “false”
delete_watched_namespaces: “false”
disable_telemetry: “false”
namespace: “bsaf-preprod-consumer”
namespace_mode: “disabled”
pgo_image_prefix: “bsaf-preprod-consumer/percona-postgresql-operator”
pgo_image_pull_policy: “Always”
pgo_image_pull_secret: “”
pgo_image_pull_secret_manifest: “”
pgo_image_tag: “1.2.0”
pgo_installation_name: “devtest”
pgo_admin_password: “examplepassword”
pgo_admin_perms: “*”
pgo_admin_role_name: “pgoadmin”
pgo_admin_username: “admin”
ccp_image_prefix: “bsaf-preprod-consumer/percona-postgresql-operator”
ccp_image_tag: “1.2.0-postgres-ha”
disable_auto_failover: “false”
db_name: “”
db_password_age_days: “0”
db_password_length: “24”
backup_storage: “default”
primary_storage: “default”
replica_storage: “default”
pgo_client_version: “4.7.1”
pgbadgerport: “10000”
sync_replication: “false”
exporterport: “9187”
scheduler_timeout: “3600”
db_password_length: “24”
db_port: “5432”
db_replicas: “0”
db_user: “testuser”
badger: “false”
metrics: “true”
backrest_storage: “default”
wal_storage: “”
pgadmin_storage: “default”
pgo_operator_namespace: “bsaf-preprod-consumer”


apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: pgo-deployer-crb
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: pgo-deployer-cr
subjects:

  • kind: ServiceAccount
    name: pgo-deployer-sa
    namespace: bsaf-preprod-consumer

apiVersion: batch/v1
kind: Job
metadata:
name: pgo-deploy
spec:
backoffLimit: 0
template:
metadata:
name: pgo-deploy
spec:
serviceAccountName: pgo-deployer-sa
restartPolicy: Never
containers:
- name: pgo-deploy
image: bsaf-preprod-consumer/percona-postgresql-operator:1.2.0-pgo-deployer
imagePullPolicy: Always
resources:
limits:
cpu: ‘2’
memory: 2Gi
requests:
cpu: ‘1’
memory: 1Gi
env:
- name: DEPLOY_ACTION
value: install
volumeMounts:
- name: deployer-conf
mountPath: “/conf”
volumes:
- name: deployer-conf
configMap:
name: pgo-deployer-cm

@Roua_Saad seems that you forgot to set the namespace during apply.
As you see in the yaml the namespace is set only in a couple of places. Could you please make sure that you specify it as well during apply?

oc apply -n bsaf-preprod-consumer -f operator.yaml –as=system:admin

@Sergey_Pronin yes I’ve deployed it using the correct namespace and the job appeared on the right project with the error provided earlier

@Roua_Saad did you run the command as I explained, specifying the namespace? If you did not, than service account is in the wrong namespace.

@Sergey_Pronin I did apply the command that you provided,but the pod of the job pgo-deployer kept failing with the following error in the logs:

@Roua_Saad there can be two reasons to that:

  1. service account was modified
  2. service account was not created or created in another namespace

I would encourage you to start from a clean slate - delete everything and ensure you set namespace correctly.

We can jump into a quick call to discuss it and see what is happening.

@Sergey_Pronin, thank you for following up, I’ll delete everything and attempt to redeploy it once more to see the outcome.