PostgreSQL operator pods fail to come up

Hello,

I have followed the PostgreSQL Operator installation procedure listed @ Generic Kubernetes installation - Percona Operator for PostgreSQL.

Note: I have installed the operator in a custom namespace(mysqldb) but made the necessary changes to the namespace name as mentioned in the instructions.

kubectl config set-context $(kubectl config current-context) --namespace=mysqldb

kubectl apply -f deploy/operator.yaml

kubectl get pods

NAME READY STATUS RESTARTS AGE

pgo-deploy-skwcd 0/1 Error 0 96s

kubectl logs pod/pgo-deploy-skwcd

nss_wrapper: user exists

nss_wrapper: group exists

nss_wrapper: environment configured

ERROR: ld.so: object ‘/usr/lib64/libnss_wrapper.so’ from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.

ERROR: ld.so: object ‘/usr/lib64/libnss_wrapper.so’ from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.

1 Like

Hey @Ravi_Kumar_Pokala ,

thanks for sharing.

  1. I can’t reproduce it and your steps look correct.
  2. You can safely ignore LD_PRELOAD errors.
  3. Can you please show the full log of the deploy pod and share its status? kubectl describe pod pgo-deploy-skwcd
  4. Which k8s flavor do you use? Anything specific about it?
1 Like

Thanks for quick response @Sergey_Pronin , please find the details below.

> kubectl version
Server Version: version.Info{Major:“1”, Minor:“20”, GitVersion:“v1.20.5”, GitCommit:“6b1d87acf3c8253c123756b9e61dac642678305f”, GitTreeState:“clean”, BuildDate:“2021-03-18T01:02:01Z”, GoVersion:“go1.15.8”, Compiler:“gc”, Platform:“linux/amd64”}

>k describe pod/pgo-deploy-skwcd
Name: pgo-deploy-skwcd
Namespace: mysqldb
Priority: 0
Service Account: pgo-deployer-sa
Node: 10.47.244.21/10.47.244.21
Start Time: Thu, 12 Jan 2023 14:13:56 -0800
Labels: controller-uid=e9d58043-0fce-4b1f-b41b-89307cb7170a
job-name=pgo-deploy
Annotations: cni.projectcalico.org/podIP:
cni.projectcalico.org/podIPs:
Status: Failed
IP: 10.20.112.165
IPs:
IP: 10.20.112.165
Controlled By: Job/pgo-deploy
Containers:
pgo-deploy:
Container ID: 8773c4869f1056746eaecdf81d335e3c0d8225ac43155178ff22db73ccb72bf7
Image: percona-postgresql-operator:1.3.0-pgo-deployer
Image ID: docker-pullable percona/percona-postgresql-operator@sha256:4f1e7292db27fcd7cbd96066d9e22c38a5cdfbefc5701d003cef8c631587401c
Port:
Host Port:
State: Terminated
Reason: Error
Exit Code: 2
Started: Thu, 12 Jan 2023 14:14:09 -0800
Finished: Thu, 12 Jan 2023 14:15:19 -0800
Ready: False
Restart Count: 0
Limits:
cpu: 2
memory: 1Gi
Requests:
cpu: 2
memory: 1Gi
Environment:
DEPLOY_ACTION: install
Mounts:
/conf from deployer-conf (rw)
/var/run/secrets/kubernetes.io/serviceaccount from pgo-deployer-sa-token-spqng (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
deployer-conf:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: pgo-deployer-cm
Optional: false
pgo-deployer-sa-token-spqng:
Type: Secret (a volume populated by a Secret)
SecretName: pgo-deployer-sa-token-spqng
Optional: false
QoS Class: Guaranteed
Node-Selectors:
Tolerations: node.kubernetes.io-not-ready:NoExecute op=Exists for 300s
node.kubernetes.io-unreachable:NoExecute op=Exists for 300s
Events:

1 Like

@Sergey_Pronin : Please find the pod description below. For some reason it this forum doesn’t let me to add full logs and complains about new users restriction of 2 links only.

>kubectl describe pod/pgo-deploy-skwcd
Name: pgo-deploy-skwcd
Namespace: mysqldb
Priority: 0
Service Account: pgo-deployer-sa
Node: 10.47.244.21/10.47.244.21
Start Time: Thu, 12 Jan 2023 14:13:56 -0800
Labels: controller-uid=e9d58043-0fce-4b1f-b41b-89307cb7170a
job-name=pgo-deploy
Annotations: cni.projectcalico.org/podIP:
cni.projectcalico.org/podIPs:
Status: Failed
IP: 10.20.112.165
IPs:
IP: 10.20.112.165
Controlled By: Job/pgo-deploy
Containers:
pgo-deploy:
Container ID: docker-8773c4869f1056746eaecdf81d335e3c0d8225ac43155178ff22db73ccb72bf7
Image: percona/percona-postgresql-operator-1.3.0-pgo-deployer
Image ID: docker-pullable-percona/percona-postgresql-operator@sha256:4f1e7292db27fcd7cbd96066d9e22c38a5cdfbefc5701d003cef8c631587401c
Port:
Host Port:
State: Terminated
Reason: Error
Exit Code: 2
Started: Thu, 12 Jan 2023 14:14:09 -0800
Finished: Thu, 12 Jan 2023 14:15:19 -0800
Ready: False
Restart Count: 0
Limits:
cpu: 2
memory: 1Gi
Requests:
cpu: 2
memory: 1Gi
Environment:
DEPLOY_ACTION: install
Mounts:
/conf from deployer-conf (rw)
/var/run/secrets/kubernetes.io/serviceaccount from pgo-deployer-sa-token-spqng (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
deployer-conf:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: pgo-deployer-cm
Optional: false
pgo-deployer-sa-token-spqng:
Type: Secret (a volume populated by a Secret)
SecretName: pgo-deployer-sa-token-spqng
Optional: false
QoS Class: Guaranteed
Node-Selectors:
Tolerations: node.kubernetes.io-not-ready:NoExecute op=Exists for 300s
node.kubernetes.io-unreachable:NoExecute op=Exists for 300s
Events:

1 Like

> kubectl version
Server Version: version.Info{Major:“1”, Minor:“20”, GitVersion:“v1.20.5”, GitCommit:“6b1d87acf3c8253c123756b9e61dac642678305f”, GitTreeState:“clean”, BuildDate:“2021-03-18T01:02:01Z”, GoVersion:“go1.15.8”, Compiler:“gc”, Platform:“linux/amd64”}

1 Like

> kubectl version
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short. Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:“1”, Minor:“25”, GitVersion:“v1.25.4”, GitCommit:“872a965c6c6526caa949f0c6ac028ef7aff3fb78”, GitTreeState:“clean”, BuildDate:“2022-11-09T13:36:36Z”, GoVersion:“go1.19.3”, Compiler:“gc”, Platform:“linux/amd64”}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:“1”, Minor:“20”, GitVersion:“v1.20.5”, GitCommit:“6b1d87acf3c8253c123756b9e61dac642678305f”, GitTreeState:“clean”, BuildDate:“2021-03-18T01:02:01Z”, GoVersion:“go1.15.8”, Compiler:“gc”, Platform:“linux/amd64”}

1 Like

Pod logs
nss_wrapper: user exists
nss_wrapper: group exists
nss_wrapper: environment configured
ERROR: ld.so: object ‘/usr/lib64/libnss_wrapper.so’ from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
ERROR: ld.so: object ‘/usr/lib64/libnss_wrapper.so’ from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
Process install dependency map
ERROR: ld.so: object ‘/usr/lib64/libnss_wrapper.so’ from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
ERROR! Unknown error when attempting to call Galaxy at ‘galaxy-ansible-com/api’: <urlopen error [Errno 99] Cannot assign requested address>
ERROR: ld.so: object ‘/usr/lib64/libnss_wrapper.so’ from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
ERROR: ld.so: object ‘/usr/lib64/libnss_wrapper.so’ from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
ERROR: ld.so: object ‘/usr/lib64/libnss_wrapper.so’ from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
|
PLAY [Deploy Crunchy PostgreSQL Operator] **************************************

TASK [Gathering Facts] *********************************************************
ok: [localhost]

TASK [pgo-preflight : include_tasks] *******************************************
included: /ansible/postgres-operator/roles/pgo-preflight/tasks/vars.yml for localhost

TASK [pgo-preflight : Check for “/conf/values.yaml”] ***************************
ok: [localhost]

TASK [pgo-preflight : fail] ****************************************************
skipping: [localhost]

TASK [pgo-preflight : include_vars] ********************************************
ok: [localhost]

TASK [pgo-preflight : fail] ****************************************************
skipping: [localhost]

TASK [pgo-preflight : Check Operating System] **********************************
changed: [localhost]

TASK [pgo-preflight : assert] **************************************************
ok: [localhost] => {
“changed”: false,
“msg”: “All assertions passed”
}

TASK [pgo-preflight : assert] **************************************************
skipping: [localhost]

TASK [pgo-preflight : assert] **************************************************
skipping: [localhost]

TASK [pgo-preflight : assert] **************************************************
ok: [localhost] => {
“changed”: false,
“msg”: “All assertions passed”
}

TASK [pgo-preflight : include_tasks] *******************************************
skipping: [localhost]

TASK [pgo-preflight : include_tasks] *******************************************
included: /ansible/postgres-operator/roles/pgo-preflight/tasks/check_kubernetes.yml for localhost

TASK [pgo-preflight : Check if the kubectl command is installed] ***************
changed: [localhost]

TASK [pgo-preflight : Ensure kubectl is installed] *****************************
ok: [localhost] => {
“changed”: false,
“msg”: “All assertions passed”
}

TASK [pgo-preflight : include_tasks] *******************************************
included: /ansible/postgres-operator/roles/pgo-preflight/tasks/check_vars.yml for localhost

TASK [pgo-preflight : Check if mandatory variables are defined] ****************
skipping: [localhost] => (item=pgo_operator_namespace)
skipping: [localhost] => (item=pgo_installation_name)
skipping: [localhost] => (item=pgo_admin_username)
skipping: [localhost] => (item=pgo_admin_role_name)
skipping: [localhost] => (item=pgo_admin_perms)
skipping: [localhost] => (item=ccp_image_prefix)
skipping: [localhost] => (item=ccp_image_tag)
skipping: [localhost] => (item=pgo_image_prefix)
skipping: [localhost] => (item=pgo_image_tag)
skipping: [localhost] => (item=disable_auto_failover)
skipping: [localhost] => (item=badger)
skipping: [localhost] => (item=metrics)
skipping: [localhost] => (item=archive_mode)
skipping: [localhost] => (item=archive_timeout)
skipping: [localhost] => (item=db_password_length)
skipping: [localhost] => (item=create_rbac)
skipping: [localhost] => (item=db_port)
skipping: [localhost] => (item=db_replicas)
skipping: [localhost] => (item=db_user)
skipping: [localhost] => (item=backrest_storage)
skipping: [localhost] => (item=backup_storage)
skipping: [localhost] => (item=primary_storage)
skipping: [localhost] => (item=replica_storage)
skipping: [localhost] => (item=pgo_client_version)
skipping: [localhost] => (item=pgbadgerport)
skipping: [localhost] => (item=exporterport)
skipping: [localhost] => (item=scheduler_timeout)
skipping: [localhost] => (item=namespace_mode)
skipping: [localhost] => (item=reconcile_rbac)

TASK [pgo-operator : Set output directory fact] ********************************
ok: [localhost]

TASK [pgo-operator : Ensure output directory exists] ***************************
changed: [localhost]

TASK [pgo-operator : include_tasks] ********************************************
skipping: [localhost] => (item=openshift_auth.yml)
skipping: [localhost] => (item=openshift.yml)

TASK [pgo-operator : include_tasks] ********************************************
included: /ansible/postgres-operator/roles/pgo-operator/tasks/kubernetes_auth.yml for localhost
included: /ansible/postgres-operator/roles/pgo-operator/tasks/kubernetes.yml for localhost

TASK [pgo-operator : Set the Kubernetes Context] *******************************
skipping: [localhost]

TASK [pgo-operator : Get Namespace Details] ************************************
changed: [localhost]

TASK [pgo-operator : Create PGO Namespace] *************************************
skipping: [localhost]

TASK [pgo-operator : Use kubectl or oc] ****************************************
ok: [localhost]

TASK [pgo-operator : include_tasks] ********************************************
included: /ansible/postgres-operator/roles/pgo-operator/tasks/namespace.yml for localhost

TASK [pgo-operator : Namespace List] *******************************************
ok: [localhost]

TASK [pgo-operator : Create Watched Namespaces] ********************************
skipping: [localhost] => (item=mysqldb)

TASK [pgo-operator : Create Watched Namespaces (Reconcile RBAC)] ***************
changed: [localhost] => (item=mysqldb)

TASK [pgo-operator : Label Watched Namespaces (Reconcile RBAC)] ****************
changed: [localhost] => (item=mysqldb)

TASK [pgo-operator : Cleanup Local Namespace Target RBAC] **********************
failed: [localhost] (item=mysqldb) => {“ansible_loop_var”: “item”, “changed”: true, “cmd”: [“kubectl”, “delete”, “role,rolebinding”, “pgo-target-role”, “-n”, “mysqldb”], “delta”: “0:00:00.127363”, “end”: “2023-01-12 22:15:16.431289”, “item”: “mysqldb”, “msg”: “non-zero return code”, “rc”: 1, “start”: “2023-01-12 22:15:16.303926”, “stderr”: “Error from server (NotFound): rolebindings-rbac-authorization-k8s-io "pgo-target-role" not found”, “stderr_lines”: [“Error from server (NotFound): rolebindings-rbac-authorization-k8s-io "pgo-target-role" not found”], “stdout”: “role.rbac.authorization.k8s-io "pgo-target-role" deleted”, “stdout_lines”: [“role.rbac.authorization.k8s-io "pgo-target-role" deleted”]}
…ignoring

TASK [pgo-operator : Create Local Namespace Target RBAC] ***********************
changed: [localhost] => (item=mysqldb)

TASK [pgo-operator : Template Local Namespace RBAC] ****************************
changed: [localhost]

TASK [pgo-operator : Cleanup Local Namespace Reconcile RBAC] *******************
changed: [localhost] => (item=mysqldb)

TASK [pgo-operator : Create Local Namespace Reconcile RBAC] ********************
changed: [localhost] => (item=mysqldb)

TASK [pgo-operator : include_tasks] ********************************************
fatal: [localhost]: FAILED! => {“reason”: “couldn’t resolve module/action ‘kubernetes.core.k8s’. This often indicates a misspelling, missing collection, or incorrect module path.\n\nThe error appears to be in ‘/ansible/postgres-operator/roles/pgo-operator/tasks/crds.yml’: line 67, column 5, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n block:\n - name: Create PerconaPGClusters CRD\n ^ here\n”}

PLAY RECAP *********************************************************************
localhost : ok=26 changed=11 unreachable=0 failed=1 skipped=10 rescued=0 ignored=1

1 Like

@Ravi_Kumar_Pokala

the problem is that for some reason kubernetes.core ansible collection is missing.

See this error:

ERROR! Unknown error when attempting to call Galaxy at ‘galaxy-ansible-com/api’: <urlopen error [Errno 99] Cannot assign requested address>

Do you block any external connections? Seems ansible galaxy is not reachable. It tries to connect to https://galaxy.ansible.com/api/.

I would also suggest you to try our PG Operator v2 (it is in tech preview) - it does not have this deployment process with ansible, but relies on regular CRDs. It is much simpler and cleaner.

Read more about v2 here: Percona Operator for PostgreSQL 2.0.0 Tech preview (2022-12-30) - Percona Operator for PostgreSQL

1 Like

@Sergey_Pronin : Thank you. Any idea when V2 (PostgreSQL 2.0.0) would be production ready?

1 Like

@Sergey_Pronin : I was able to fix the issue, it was because of http_proxy settings in my ENV. Now I am able to run the operator. But I am not able to create PG CLusters. I have followed the instructions below. I don’t see pods for the cluster.

> kubectl apply -f deploy/cr.yaml

>kubeclt logs PerconaPGCluster/cluster1
error: no kind “PerconaPGCluster” is registered for version “pg.percona.com/v1” in scheme “pkg/scheme/scheme.go:28”

>kubectl describe PerconaPGCluster/cluster1
Name: cluster1
Namespace: mysqldb
Labels: pgo-version=1.3.0
Annotations:
API Version: pg.percona.com/v1
Kind: PerconaPGCluster
Metadata:
Creation Timestamp: 2023-01-18T05:39:05Z
Generation: 1
Managed Fields:
API Version: pg.percona.com/v1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.:
f:kubectl.kubernetes.io/last-applied-configuration:
f:labels:
.:
f:pgo-version:
f:spec:
.:
f:backup:
f:database:
f:disableAutofail:
f:keepBackups:
f:keepData:
f:pause:
f:pgBadger:
f:pgBouncer:
f:pgPrimary:
f:pgReplicas:
f:pmm:
f:port:
f:standby:
f:tlsOnly:
f:upgradeOptions:
f:user:
Manager: kubectl-client-side-apply
Operation: Update
Time: 2023-01-18T05:39:05Z
Resource Version: 336927727
UID: 61597708-0c85-4a1e-be4b-bc6e1cb2804b
Spec:
Backup:
Affinity:
Anti Affinity Type: preferred
Backrest Repo Image: percona/percona-postgresql-operator:1.3.0-ppg14-pgbackrest-repo
Image: percona/percona-postgresql-operator:1.3.0-ppg14-pgbackrest
Resources:
Limits:
Cpu: 1
Memory: 64Mi
Requests:
Cpu: 200m
Memory: 48Mi
Schedule:
Keep: 3
Name: sat-night-backup
Schedule: 0 0 * * 6
Storage: local
Type: full
Volume Spec:
Accessmode: ReadWriteOnce
Size: 1G
Storageclass:
Storagetype: dynamic
Database: pgdb
Disable Autofail: false
Keep Backups: true
Keep Data: true
Pause: false
Pg Badger:
Enabled: false
Image: percona/percona-postgresql-operator:1.3.0-ppg14-pgbadger
Port: 10000
Pg Bouncer:
Affinity:
Anti Affinity Type: preferred
Expose:
Service Type: ClusterIP
Image: percona/percona-postgresql-operator:1.3.0-ppg14-pgbouncer
Resources:
Limits:
Cpu: 2
Memory: 512Mi
Requests:
Cpu: 1
Memory: 128Mi
Size: 3
Pg Primary:
Affinity:
Anti Affinity Type: preferred
Expose:
Service Type: ClusterIP
Image: percona/percona-postgresql-operator:1.3.0-ppg14-postgres-ha
Resources:
Limits:
Cpu: 500m
Memory: 256Mi
Requests:
Cpu: 500m
Memory: 256Mi
Tolerations:
Volume Spec:
Accessmode: ReadWriteOnce
Size: 1G
Storageclass:
Storagetype: dynamic
Pg Replicas:
Hot Standby:
Enable Sync Standby: false
Expose:
Service Type: ClusterIP
Resources:
Limits:
Cpu: 500m
Memory: 256Mi
Requests:
Cpu: 500m
Memory: 256Mi
Size: 2
Volume Spec:
Accessmode: ReadWriteOnce
Size: 1G
Storageclass:
Storagetype: dynamic
Pmm:
Enabled: false
Image: percona/pmm-client:2.29.0
Pmm Secret: cluster1-pmm-secret
Resources:
Limits:
Cpu: 1
Memory: 400M
Requests:
Cpu: 500m
Memory: 200M
Server Host: monitoring-service
Server User: admin
Port: 5432
Standby: false
Tls Only: false
Upgrade Options:
Apply: disabled
Schedule: 0 4 * * *
Version Service Endpoint: https://check.percona.com
User: pguser
Events:

1 Like

Hey @Ravi_Kumar_Pokala ,

pg v2 will go GA somewhere Q2-Q3 this year.

As for your issue with pgcluster - do you have anything in the logs of the operator? Any deployment objects were created?

1 Like

Hi, I’m also willing to try v2 of pgsql operator, the thing is, I can’t install it with Helm, the v2 is not available at Percona Helm Charts | percona-helm-charts

1 Like

@Sergey_Pronin : no deployment object is created, below are the logs from operator.

>kubectl logs pod/postgres-operator-6c7cd57d65-wshgj
Defaulted container “apiserver” out of: apiserver, operator, scheduler, event
time=“2023-01-18T05:27:23Z” level=info msg=“debug flag set to false”
time=“2023-01-18T05:27:28Z” level=info msg=“postgres-operator apiserver starts” func=“main.main()” file=“/go/src/github.com/percona/percona-postgresql-operator/cmd/apiserver/main.go:111” version=1.3.0
time=“2023-01-18T05:27:28Z” level=info msg=“Pgo Namespace is [mysqldb]” func=“github.com/percona/percona-postgresql-operator/internal/apiserver.Initialize()” file=“/go/src/github.com/percona/percona-postgresql-operator/internal/apiserver/root.go:100” version=1.3.0
time=“2023-01-18T05:27:28Z” level=info msg=“InstallationName is [devtest]” func=“github.com/percona/percona-postgresql-operator/internal/apiserver.Initialize()” file=“/go/src/github.com/percona/percona-postgresql-operator/internal/apiserver/root.go:107” version=1.3.0
time=“2023-01-18T05:27:28Z” level=info msg=“apiserver starts” func=“github.com/percona/percona-postgresql-operator/internal/apiserver.Initialize()” file=“/go/src/github.com/percona/percona-postgresql-operator/internal/apiserver/root.go:119” version=1.3.0
time=“2023-01-18T05:27:28Z” level=info msg=“loading PermMap with 56 Permissions\n” func=“github.com/percona/percona-postgresql-operator/internal/apiserver.initializePerms()” file=“/go/src/github.com/percona/percona-postgresql-operator/internal/apiserver/perms.go:179” version=1.3.0
time=“2023-01-18T05:27:28Z” level=info msg=“Config: "pgo-config" ConfigMap found, using config files from the configmap” func=“github.com/percona/percona-postgresql-operator/internal/config.initialize()” file=“/go/src/github.com/percona/percona-postgresql-operator/internal/config/pgoconfig.go:752” version=1.3.0
I0118 05:27:30.012386 1 request.go:665] Waited for 1.016857978s due to client-side throttling, not priority and fairness, request: GET:https://10.21.0.1:443/apis/policy/v1beta1
time=“2023-01-18T05:27:31Z” level=info msg=“default instance memory set to [128Mi]” func=“github.com/percona/percona-postgresql-operator/internal/config.(*PgoConfig).Validate()” file=“/go/src/github.com/percona/percona-postgresql-operator/internal/config/pgoconfig.go:394” version=1.3.0
time=“2023-01-18T05:27:31Z” level=info msg=“default pgbackrest repository memory set to [48Mi]” func=“github.com/percona/percona-postgresql-operator/internal/config.(*PgoConfig).Validate()” file=“/go/src/github.com/percona/percona-postgresql-operator/internal/config/pgoconfig.go:400” version=1.3.0
time=“2023-01-18T05:27:31Z” level=info msg=“default pgbouncer memory set to [24Mi]” func=“github.com/percona/percona-postgresql-operator/internal/config.(*PgoConfig).Validate()” file=“/go/src/github.com/percona/percona-postgresql-operator/internal/config/pgoconfig.go:406” version=1.3.0
time=“2023-01-18T05:27:31Z” level=info msg=“BasicAuth is true” func=“github.com/percona/percona-postgresql-operator/internal/apiserver.initConfig()” file=“/go/src/github.com/percona/percona-postgresql-operator/internal/apiserver/root.go:190” version=1.3.0
time=“2023-01-18T05:27:31Z” level=info msg=“Namespace operating mode is ‘disabled’” func=“github.com/percona/percona-postgresql-operator/internal/apiserver.Initialize()” file=“/go/src/github.com/percona/percona-postgresql-operator/internal/apiserver/root.go:151” version=1.3.0
time=“2023-01-18T05:27:31Z” level=info msg=“pgo.tls Secret found in namespace mysqldb” func=“github.com/percona/percona-postgresql-operator/internal/apiserver.WriteTLSCert()” file=“/go/src/github.com/percona/percona-postgresql-operator/internal/apiserver/root.go:418” version=1.3.0
time=“2023-01-18T05:27:31Z” level=info msg=“cert key data len is 558” func=“github.com/percona/percona-postgresql-operator/internal/apiserver.WriteTLSCert()” file=“/go/src/github.com/percona/percona-postgresql-operator/internal/apiserver/root.go:419” version=1.3.0
time=“2023-01-18T05:27:31Z” level=info msg=“private key data len is 241” func=“github.com/percona/percona-postgresql-operator/internal/apiserver.WriteTLSCert()” file=“/go/src/github.com/percona/percona-postgresql-operator/internal/apiserver/root.go:425” version=1.3.0
time=“2023-01-18T05:27:31Z” level=info msg=“listening on port 8443” func=“main.main()” file=“/go/src/github.com/percona/percona-postgresql-operator/cmd/apiserver/main.go:182” version=1.3.0

1 Like

@Sergey_Pronin : I am still looking for help on this. any insight into the issue?

Hey @Ravi_Kumar_Pokala ,

I still can’t reproduce it. Please give me the exact steps to reproduce.

  1. I deploy the operator (wait for it to come up)
  2. I deploy the cluster

I’m using default yamls, not changes whatsoever. What are your steps?

Also please share logs from the operator container, not apiserver.

@Sergey_Pronin : I have followed steps listed below only change to defaults was

  1. Namespace
  2. Storage Class.

Please find the steps and errors from the operator container below.

>kubectl apply -f deploy/operator.yaml
serviceaccount/pgo-deployer-sa created
clusterrole.rbac.authorization.k8s.io/pgo-deployer-cr created
configmap/pgo-deployer-cm created
clusterrolebinding.rbac.authorization.k8s.io/pgo-deployer-crb created
job.batch/pgo-deploy created

>k logs pod/postgres-operator-54cb89b45b-6cp8z -c operator
time=“2023-01-31T22:55:22Z” level=info msg=“debug flag set to false” func=“main.main()” file=“main.go:55” version=1.3.0
time=“2023-01-31T22:55:27Z” level=info msg=“CRUNCHY_DEBUG flag set to false” func=“github.com/percona/percona-postgresql-operator/internal/operator.Initialize()” file=“/go/src/github.com/percona/percona-postgresql-operator/internal/operator/common.go:104” version=1.3.0
time=“2023-01-31T22:55:27Z” level=info msg=“NAMESPACE percona-pgo” func=“github.com/percona/percona-postgresql-operator/internal/operator.Initialize()” file=“/go/src/github.com/percona/percona-postgresql-operator/internal/operator/common.go:108” version=1.3.0
time=“2023-01-31T22:55:27Z” level=info msg=“InstallationName devtest” func=“github.com/percona/percona-postgresql-operator/internal/operator.Initialize()” file=“/go/src/github.com/percona/percona-postgresql-operator/internal/operator/common.go:111” version=1.3.0
time=“2023-01-31T22:55:27Z” level=info msg=“Config: "pgo-config" ConfigMap found, using config files from the configmap” func=“github.com/percona/percona-postgresql-operator/internal/config.initialize()” file=“/go/src/github.com/percona/percona-postgresql-operator/internal/config/pgoconfig.go:752” version=1.3.0
I0131 22:55:28.900999 1 request.go:665] Waited for 1.018727122s due to client-side throttling, not priority and fairness, request: GET:https://10.21.0.1:443/apis/autoscaling/v2beta2
time=“2023-01-31T22:55:30Z” level=info msg=“default instance memory set to [128Mi]” func=“github.com/percona/percona-postgresql-operator/internal/config.(*PgoConfig).Validate()” file=“/go/src/github.com/percona/percona-postgresql-operator/internal/config/pgoconfig.go:394” version=1.3.0
time=“2023-01-31T22:55:30Z” level=info msg=“default pgbackrest repository memory set to [48Mi]” func=“github.com/percona/percona-postgresql-operator/internal/config.(*PgoConfig).Validate()” file=“/go/src/github.com/percona/percona-postgresql-operator/internal/config/pgoconfig.go:400” version=1.3.0
time=“2023-01-31T22:55:30Z” level=info msg=“default pgbouncer memory set to [24Mi]” func=“github.com/percona/percona-postgresql-operator/internal/config.(*PgoConfig).Validate()” file=“/go/src/github.com/percona/percona-postgresql-operator/internal/config/pgoconfig.go:406” version=1.3.0
time=“2023-01-31T22:55:30Z” level=info msg=“EventTCPAddress set to localhost:4150” func=“github.com/percona/percona-postgresql-operator/internal/operator.Initialize()” file=“/go/src/github.com/percona/percona-postgresql-operator/internal/operator/common.go:161” version=1.3.0
I0131 22:55:30.520881 1 shared_informer.go:240] Waiting for caches to sync for namespace
I0131 22:55:30.621261 1 shared_informer.go:247] Caches are synced for namespace
time=“2023-01-31T22:55:30Z” level=info msg=“PostgreSQL Operator initialized and running, waiting for signal to exit” func=“main.main()” file=“main.go:125” version=1.3.0
I0131 22:55:30.744741 1 shared_informer.go:240] Waiting for caches to sync for percona-pgo
I0131 22:55:30.845820 1 shared_informer.go:247] Caches are synced for percona-pgo
time=“2023-01-31T22:55:35Z” level=error msg=“reconcile perocnapgclusters: list perconapgclusters: perconapgclusters.pg.percona.com is forbidden: User "system:serviceaccount:percona-pgo:postgres-operator" cannot list resource "perconapgclusters" in API group "pg.percona.com" in the namespace "percona-pgo"” func=“github.com/percona/percona-postgresql-operator/percona/controllers/pgc.(*Controller).reconcilePerconaPG()” file=“/go/src/github.com/percona/percona-postgresql-operator/percona/controllers/pgc/pgc.go:310” version=1.3.0
time=“2023-01-31T22:55:40Z” level=error msg=“reconcile perocnapgclusters: list perconapgclusters: perconapgclusters.pg.percona.com is forbidden: User "system:serviceaccount:percona-pgo:postgres-operator" cannot list resource "perconapgclusters" in API group "pg.percona.com" in the namespace "percona-pgo"” func=“github.com/percona/percona-postgresql-operator/percona/controllers/pgc.(*Controller).reconcilePerconaPG()” file=“/go/src/github.com/percona/percona-postgresql-operator/percona/controllers/pgc/pgc.go:310” version=1.3.0
time=“2023-01-31T22:55:45Z” level=error msg=“reconcile perocnapgclusters: list perconapgclusters: perconapgclusters.pg.percona.com is forbidden: User "system:serviceaccount:percona-pgo:postgres-operator" cannot list resource "perconapgclusters" in API group "pg.percona.com" in the namespace "percona-pgo"” func=“github.com/percona/percona-postgresql-operator/percona/controllers/pgc.(*Controller).reconcilePerconaPG()” file=“/go/src/github.com/percona/percona-postgresql-operator/percona/controllers/pgc/pgc.go:310” version=1.3.0
time=“2023-01-31T22:55:50Z” level=error msg=“reconcile perocnapgclusters: list perconapgclusters: perconapgclusters.pg.percona.com is forbidden: User "system:serviceaccount:percona-pgo:postgres-operator" cannot list resource "perconapgclusters" in API group "pg.percona.com" in the namespace "percona-pgo"” func=“github.com/percona/percona-postgresql-operator/percona/controllers/pgc.(*Controller).reconcilePerconaPG()” file=“/go/src/github.com/percona/percona-postgresql-operator/percona/controllers/pgc/pgc.go:310” version=1.3.0
time=“2023-01-31T22:55:55Z” level=error msg=“reconcile perocnapgclusters: list perconapgclusters: perconapgclusters.pg.percona.com is forbidden: User "system:serviceaccount:percona-pgo:postgres-operator" cannot list resource "perconapgclusters" in API group "pg.percona.com" in the namespace "percona-pgo"” func=“github.com/percona/percona-postgresql-operator/percona/controllers/pgc.(*Controller).reconcilePerconaPG()” file=“/go/src/github.com/percona/percona-postgresql-operator/percona/controllers/pgc/pgc.go:310” version=1.3.0
time=“2023-01-31T22:56:00Z” level=error msg=“reconcile perocnapgclusters: list perconapgclusters: perconapgclusters.pg.percona.com is forbidden: User "system:serviceaccount:percona-pgo:postgres-operator" cannot list resource "perconapgclusters" in API group "pg.percona.com" in the namespace "percona-pgo"” func=“github.com/percona/percona-postgresql-operator/percona/controllers/pgc.(*Controller).reconcilePerconaPG()” file=“/go/src/github.com/percona/percona-postgresql-operator/percona/controllers/pgc/pgc.go:310” version=1.3.0
time=“2023-01-31T22:56:05Z” level=error msg=“reconcile perocnapgclusters: list perconapgclusters: perconapgclusters.pg.percona.com is forbidden: User "system:serviceaccount:percona-pgo:postgres-operator" cannot list resource "perconapgclusters" in API group "pg.percona.com" in the namespace "percona-pgo"” func=“github.com/percona/percona-postgresql-operator/percona/controllers/pgc.(*Controller).reconcilePerconaPG()” file=“/go/src/github.com/percona/percona-postgresql-operator/percona/controllers/pgc/pgc.go:310” version=1.3.0
time=“2023-01-31T22:56:10Z” level=error msg=“reconcile perocnapgclusters: list perconapgclusters: perconapgclusters.pg.percona.com is forbidden: User "system:serviceaccount:percona-pgo:postgres-operator" cannot list resource "perconapgclusters" in API group "pg.percona.com" in the namespace "percona-pgo"” func=“github.com/percona/percona-postgresql-operator/percona/controllers/pgc.(*Controller).reconcilePerconaPG()” file=“/go/src/github.com/percona/percona-postgresql-operator/percona/controllers/pgc/pgc.go:310” version=1.3.0
time=“2023-01-31T22:56:15Z” level=error msg=“reconcile perocnapgclusters: list perconapgclusters: perconapgclusters.pg.percona.com is forbidden: User "system:serviceaccount:percona-pgo:postgres-operator" cannot list resource "perconapgclusters" in API group "pg.percona.com" in the namespace "percona-pgo"” func=“github.com/percona/percona-postgresql-operator/percona/controllers/pgc.(*Controller).reconcilePerconaPG()” file=“/go/src/github.com/percona/percona-postgresql-operator/percona/controllers/pgc/pgc.go:310” version=1.3.0
time=“2023-01-31T22:56:20Z” level=error msg=“reconcile perocnapgclusters: list perconapgclusters: perconapgclusters.pg.percona.com is forbidden: User "system:serviceaccount:percona-pgo:postgres-operator" cannot list resource "perconapgclusters" in API group "pg.percona.com" in the namespace "percona-pgo"” func=“github.com/percona/percona-postgresql-operator/percona/controllers/pgc.(*Controller).reconcilePerconaPG()” file=“/go/src/github.com/percona/percona-postgresql-operator/percona/controllers/pgc/pgc.go:310” version=1.3.0
time=“2023-01-31T22:56:26Z” level=error msg=“reconcile perocnapgclusters: list perconapgclusters: perconapgclusters.pg.percona.com is forbidden: User "system:serviceaccount:percona-pgo:postgres-operator" cannot list resource "perconapgclusters" in API group "pg.percona.com" in the namespace "percona-pgo"” func=“github.com/percona/percona-postgresql-operator/percona/controllers/pgc.(*Controller).reconcilePerconaPG()” file=“/go/src/github.com/percona/percona-postgresql-operator/percona/controllers/pgc/pgc.go:310” version=1.3.0
time=“2023-01-31T22:56:30Z” level=error msg=“operator is unable to reconcile RBAC resource: roles.rbac.authorization.k8s.io "pgo-target-role" is forbidden: user "system:serviceaccount:percona-pgo:postgres-operator" (groups=["system:serviceaccounts" "system:serviceaccounts:percona-pgo" "system:authenticated"]) is attempting to grant RBAC permissions not currently held:\n{APIGroups:[""], Resources:["certificates"], Verbs:["get" "list" "watch" "create" "patch" "update" "delete" "deletecollection"]}\n{APIGroups:[""], Resources:["configmaps"], Verbs:["watch" "patch" "deletecollection"]}\n{APIGroups:[""], Resources:["endpoints"], Verbs:["get" "list" "watch" "create" "patch" "update" "delete" "deletecollection"]}\n{APIGroups:[""], Resources:["issuers"], Verbs:["get" "list" "watch" "create" "patch" "update" "delete" "deletecollection"]}\n{APIGroups:[""], Resources:["persistentvolumeclaims"], Verbs:["get" "list" "watch" "create" "patch" "update" "delete" "deletecollection"]}\n{APIGroups:[""], Resources:["pods"], Verbs:["get" "list" "watch" "create" "patch" "update" "delete" "deletecollection"]}\n{APIGroups:[""], Resources:["pods/exec"], Verbs:["get" "list" "watch" "create" "patch" "update" "delete" "deletecollection"]}\n{APIGroups:[""], Resources:["pods/log"], Verbs:["get" "list" "watch"]}\n{APIGroups:[""], Resources:["secrets"], Verbs:["watch" "patch" "deletecollection"]}\n{APIGroups:[""], Resources:["services"], Verbs:["get" "list" "watch" "create" "patch" "update" "delete" "deletecollection"]}\n{APIGroups:["apps"], Resources:["deployments"], Verbs:["get" "list" "watch" "create" "patch" "update" "delete" "deletecollection"]}\n{APIGroups:["apps"], Resources:["replicasets"], Verbs:["get" "list" "watch" "create" "patch" "update" "delete" "deletecollection"]}\n{APIGroups:["batch"], Resources:["jobs"], Verbs:["get" "list" "watch" "create" "patch" "update" "delete" "deletecollection"]}\n{APIGroups:["cert-manager.io"], Resources:["certificates"], Verbs:["get" "list" "watch" "create" "patch" "update" "delete" "deletecollection"]}\n{APIGroups:["cert-manager.io"], Resources:["issuers"], Verbs:["get" "list" "watch" "create" "patch" "update" "delete" "deletecollection"]}\n{APIGroups:["certmanager.k8s.io"], Resources:["certificates"], Verbs:["get" "list" "watch" "create" "patch" "update" "delete" "deletecollection"]}\n{APIGroups:["certmanager.k8s.io"], Resources:["issuers"], Verbs:["get" "list" "watch" "create" "patch" "update" "delete" "deletecollection"]}\n{APIGroups:["pg.percona.com"], Resources:["perconapgclusters"], Verbs:["get" "list" "watch" "create" "patch" "update" "delete" "deletecollection"]}\n{APIGroups:["pg.percona.com"], Resources:["perconapgclusters/status"], Verbs:["get" "list" "watch" "create" "patch" "update" "delete" "deletecollection"]}\n{APIGroups:["pg.percona.com"], Resources:["pgclusters"], Verbs:["get" "list" "watch" "create" "patch" "update" "delete" "deletecollection"]}\n{APIGroups:["pg.percona.com"], Resources:["pgpolicies"], Verbs:["get" "list" "watch" "create" "patch" "update" "delete" "deletecollection"]}\n{APIGroups:["pg.percona.com"], Resources:["pgreplicas"], Verbs:["get" "list" "watch" "create" "patch" "update" "delete" "deletecollection"]}\n{APIGroups:["pg.percona.com"], Resources:["pgtasks"], Verbs:["get" "list" "watch" "create" "patch" "update" "delete" "deletecollection"]}” func=“github.com/percona/percona-postgresql-operator/internal/controller/manager.(*ControllerManager).reconcileRoles()” file=“/go/src/github.com/percona/percona-postgresql-operator/internal/controller/manager/rbac.go:95” version=1.3.0
time=“2023-01-31T22:56:30Z” level=error msg=“operator is unable to reconcile RBAC resource: rolebindings.rbac.authorization.k8s.io "pgo-target-role-binding" is forbidden: user "system:serviceaccount:percona-pgo:postgres-operator" (groups=["system:serviceaccounts" "system:serviceaccounts:percona-pgo" "system:authenticated"]) is attempting to grant RBAC permissions not currently held:\n{APIGroups:[""], Resources:["configmaps"], Verbs:["watch" "patch" "deletecollection"]}\n{APIGroups:[""], Resources:["endpoints"], Verbs:["get" "list" "watch" "create" "patch" "update" "delete" "deletecollection"]}\n{APIGroups:[""], Resources:["persistentvolumeclaims"], Verbs:["get" "list" "watch" "create" "patch" "update" "delete" "deletecollection"]}\n{APIGroups:[""], Resources:["pods"], Verbs:["get" "list" "watch" "create" "patch" "update" "delete" "deletecollection"]}\n{APIGroups:[""], Resources:["pods/exec"], Verbs:["get" "list" "watch" "create" "patch" "update" "delete" "deletecollection"]}\n{APIGroups:[""], Resources:["pods/log"], Verbs:["get" "list" "watch"]}\n{APIGroups:[""], Resources:["secrets"], Verbs:["watch" "patch" "deletecollection"]}\n{APIGroups:[""], Resources:["services"], Verbs:["get" "list" "watch" "create" "patch" "update" "delete" "deletecollection"]}\n{APIGroups:["apps"], Resources:["deployments"], Verbs:["get" "list" "watch" "create" "patch" "update" "delete" "deletecollection"]}\n{APIGroups:["apps"], Resources:["replicasets"], Verbs:["get" "list" "watch" "create" "patch" "update" "delete" "deletecollection"]}\n{APIGroups:["batch"], Resources:["jobs"], Verbs:["get" "list" "watch" "create" "patch" "update" "delete" "deletecollection"]}\n{APIGroups:["crunchydata.com"], Resources:["pgclusters"], Verbs:["get" "list" "watch" "create" "patch" "update" "delete" "deletecollection"]}\n{APIGroups:["crunchydata.com"], Resources:["pgpolicies"], Verbs:["get" "list" "watch" "create" "patch" "update" "delete" "deletecollection"]}\n{APIGroups:["crunchydata.com"], Resources:["pgreplicas"], Verbs:["get" "list" "watch" "create" "patch" "update" "delete" "deletecollection"]}\n{APIGroups:["crunchydata.com"], Resources:["pgtasks"], Verbs:["get" "list" "watch" "create" "patch" "update" "delete" "deletecollection"]}” func=“github.com/percona/percona-postgresql-operator/internal/controller/manager.(*ControllerManager).reconcileRoleBindings()” file=“/go/src/github.com/percona/percona-postgresql-operator/internal/controller/manager/rbac.go:112” version=1.3.0
time=“2023-01-31T22:56:30Z” level=error msg=“Controller Manager: Controller Group for namespace percona-pgo does not have the required list privileges for resource pgtasks in the pg.percona.com API” func=“github.com/percona/percona-postgresql-operator/internal/controller/manager.(*ControllerManager).hasListerPrivs()” file=“/go/src/github.com/percona/percona-postgresql-operator/internal/controller/manager/controllermanager.go:357” version=1.3.0
time=“2023-01-31T22:56:30Z” level=error msg=“Controller Manager: Controller Group for namespace percona-pgo does not have the required list privileges for resource pgclusters in the pg.percona.com API” func=“github.com/percona/percona-postgresql-operator/internal/controller/manager.(*ControllerManager).hasListerPrivs()” file=“/go/src/github.com/percona/percona-postgresql-operator/internal/controller/manager/controllermanager.go:357” version=1.3.0
time=“2023-01-31T22:56:30Z” level=error msg=“Controller Manager: Controller Group for namespace percona-pgo does not have the required list privileges for resource pgreplicas in the pg.percona.com API” func=“github.com/percona/percona-postgresql-operator/internal/controller/manager.(*ControllerManager).hasListerPrivs()” file=“/go/src/github.com/percona/percona-postgresql-operator/internal/controller/manager/controllermanager.go:357” version=1.3.0
time=“2023-01-31T22:56:30Z” level=error msg=“Controller Manager: Controller Group for namespace percona-pgo does not have the required list privileges for resource pgpolicies in the pg.percona.com API” func=“github.com/percona/percona-postgresql-operator/internal/controller/manager.(*ControllerManager).hasListerPrivs()” file=“/go/src/github.com/percona/percona-postgresql-operator/internal/controller/manager/controllermanager.go:357” version=1.3.0
time=“2023-01-31T22:56:30Z” level=error msg=“Controller Manager: Controller Group for namespace percona-pgo does not have the required list privileges for resource pods in the Core API” func=“github.com/percona/percona-postgresql-operator/internal/controller/manager.(*ControllerManager).hasListerPrivs()” file=“/go/src/github.com/percona/percona-postgresql-operator/internal/controller/manager/controllermanager.go:370” version=1.3.0
time=“2023-01-31T22:56:30Z” level=error msg=“Controller Manager: Controller Group for namespace percona-pgo does not have the required list privileges for resource jobs in the Batch API” func=“github.com/percona/percona-postgresql-operator/internal/controller/manager.(*ControllerManager).hasListerPrivs()” file=“/go/src/github.com/percona/percona-postgresql-operator/internal/controller/manager/controllermanager.go:382” version=1.3.0
time=“2023-01-31T22:56:30Z” level=error msg=“Namespace Controller: error syncing Namespace ‘percona-pgo’: Controller Manager: removing the running controller group for namespace percona-pgo because it no longer has the required privs, will attempt to restart on the next ns refresh interval” func=“github.com/percona/percona-postgresql-operator/internal/controller/namespace.(*Controller).processNextWorkItem()” file=“/go/src/github.com/percona/percona-postgresql-operator/internal/controller/namespace/namespacecontroller.go:151” version=1.3.0

time=“2023-01-31T23:02:30Z” level=error msg=“operator is unable to reconcile RBAC resource: rolebindings.rbac.authorization.k8s.io "pgo-target-role-binding" is forbidden: user "system:serviceaccount:percona-pgo:postgres-operator" (groups=["system:serviceaccounts" "system:serviceaccounts:percona-pgo" "system:authenticated"]) is attempting to grant RBAC permissions not currently held:\n{APIGroups:[""], Resources:["configmaps"], Verbs:["watch" "patch" "deletecollection"]}\n{APIGroups:[""], Resources:["endpoints"], Verbs:["get" "list" "watch" "create" "patch" "update" "delete" "deletecollection"]}\n{APIGroups:[""], Resources:["persistentvolumeclaims"], Verbs:["get" "list" "watch" "create" "patch" "update" "delete" "deletecollection"]}\n{APIGroups:[""], Resources:["pods"], Verbs:["get" "list" "watch" "create" "patch" "update" "delete" "deletecollection"]}\n{APIGroups:[""], Resources:["pods/exec"], Verbs:["get" "list" "watch" "create" "patch" "update" "delete" "deletecollection"]}\n{APIGroups:[""], Resources:["pods/log"], Verbs:["get" "list" "watch"]}\n{APIGroups:[""], Resources:["secrets"], Verbs:["watch" "patch" "deletecollection"]}\n{APIGroups:[""], Resources:["services"], Verbs:["get" "list" "watch" "create" "patch" "update" "delete" "deletecollection"]}\n{APIGroups:["apps"], Resources:["deployments"], Verbs:["get" "list" "watch" "create" "patch" "update" "delete" "deletecollection"]}\n{APIGroups:["apps"], Resources:["replicasets"], Verbs:["get" "list" "watch" "create" "patch" "update" "delete" "deletecollection"]}\n{APIGroups:["batch"], Resources:["jobs"], Verbs:["get" "list" "watch" "create" "patch" "update" "delete" "deletecollection"]}\n{APIGroups:["crunchydata.com"], Resources:["pgclusters"], Verbs:["get" "list" "watch" "create" "patch" "update" "delete" "deletecollection"]}\n{APIGroups:["crunchydata.com"], Resources:["pgpolicies"], Verbs:["get" "list" "watch" "create" "patch" "update" "delete" "deletecollection"]}\n{APIGroups:["crunchydata.com"], Resources:["pgreplicas"], Verbs:["get" "list" "watch" "create" "patch" "update" "delete" "deletecollection"]}\n{APIGroups:["crunchydata.com"], Resources:["pgtasks"], Verbs:["get" "list" "watch" "create" "patch" "update" "delete" "deletecollection"]}” func=“github.com/percona/percona-postgresql-operator/internal/controller/manager.(*ControllerManager).reconcileRoleBindings()” file=“/go/src/github.com/percona/percona-postgresql-operator/internal/controller/manager/rbac.go:112” version=1.3.0
time=“2023-01-31T23:02:30Z” level=error msg=“Controller Manager: Controller Group for namespace percona-pgo does not have the required list privileges for resource pgtasks in the pg.percona.com API” func=“github.com/percona/percona-postgresql-operator/internal/controller/manager.(*ControllerManager).hasListerPrivs()” file=“/go/src/github.com/percona/percona-postgresql-operator/internal/controller/manager/controllermanager.go:357” version=1.3.0
time=“2023-01-31T23:02:30Z” level=error msg=“Controller Manager: Controller Group for namespace percona-pgo does not have the required list privileges for resource pgclusters in the pg.percona.com API” func=“github.com/percona/percona-postgresql-operator/internal/controller/manager.(*ControllerManager).hasListerPrivs()” file=“/go/src/github.com/percona/percona-postgresql-operator/internal/controller/manager/controllermanager.go:357” version=1.3.0
time=“2023-01-31T23:02:30Z” level=error msg=“Controller Manager: Controller Group for namespace percona-pgo does not have the required list privileges for resource pgreplicas in the pg.percona.com API” func=“github.com/percona/percona-postgresql-operator/internal/controller/manager.(*ControllerManager).hasListerPrivs()” file=“/go/src/github.com/percona/percona-postgresql-operator/internal/controller/manager/controllermanager.go:357” version=1.3.0
time=“2023-01-31T23:02:30Z” level=error msg=“Controller Manager: Controller Group for namespace percona-pgo does not have the required list privileges for resource pgpolicies in the pg.percona.com API” func=“github.com/percona/percona-postgresql-operator/internal/controller/manager.(*ControllerManager).hasListerPrivs()” file=“/go/src/github.com/percona/percona-postgresql-operator/internal/controller/manager/controllermanager.go:357” version=1.3.0
time=“2023-01-31T23:02:30Z” level=error msg=“Controller Manager: Controller Group for namespace percona-pgo does not have the required list privileges for resource pods in the Core API” func=“github.com/percona/percona-postgresql-operator/internal/controller/manager.(*ControllerManager).hasListerPrivs()” file=“/go/src/github.com/percona/percona-postgresql-operator/internal/controller/manager/controllermanager.go:370” version=1.3.0
time=“2023-01-31T23:02:30Z” level=error msg=“Controller Manager: Controller Group for namespace percona-pgo does not have the required list privileges for resource jobs in the Batch API” func=“github.com/percona/percona-postgresql-operator/internal/controller/manager.(*ControllerManager).hasListerPrivs()” file=“/go/src/github.com/percona/percona-postgresql-operator/internal/controller/manager/controllermanager.go:382” version=1.3.0
time=“2023-01-31T23:02:30Z” level=error msg=“Namespace Controller: error syncing Namespace ‘percona-pgo’: Controller Manager: cannot start controller group for namespace percona-pgo because it does not have the required privs, will attempt to start on the next ns refresh interval” func=“github.com/percona/percona-postgresql-operator/internal/controller/namespace.(*Controller).processNextWorkItem()” file=“/go/src/github.com/percona/percona-postgresql-operator/internal/controller/namespace/namespacecontroller.go:151” version=1.3.0

@Ravi_Kumar_Pokala please share the exact commands and YAML manifests that you used.
Previously you were talking about mysqldb namespace, now in the error log I see percona-pgo. This is a bit confusing.

BTW, which storage class are you using and what can you tell about it?

@Sergey_Pronin : Sorry for the confusion. I tried it in a fresh new namespace(percona-pgo) to rule out any issues with “mysqldb” namespace which was created long ago. but ran into the same problem.

We have a storage class(es) that supports dynamic provisioning of the volumes on PVC creation. Dynamic Provisioning and Storage Classes in Kubernetes | Kubernetes

I had used this storage class before with the other operators (including percona-xtradb-operator) and it works fine with them.

Please find the command below and the operator.yaml file is attached as txt file.

>kubectl apply -f deploy/operator.yaml
serviceaccount/pgo-deployer-sa created
clusterrole.rbac.authorization.k8s.io/pgo-deployer-cr created
configmap/pgo-deployer-cm created
clusterrolebinding.rbac.authorization.k8s.io/pgo-deployer-crb created
job.batch/pgo-deploy created

operator-yaml.txt (6.1 KB)

Hello @Ravi_Kumar_Pokala ,

interesting. Seems that postgres-operator service account does not have enough permissions.

Can you please check if pgo-target-role was created?

kubectl -n YOURNAMESPACE get role pgo-target-role -o yaml

Also, do you see any errors now in the deployer? As per all the indications it seems that some roles are not properly created, not allowing operator to see the Custom Resources.

kubectl -n percona-pgo get role pgo-target-role -o yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
creationTimestamp: “2023-01-31T23:17:07Z”
labels:
vendor: crunchydata
name: pgo-target-role
namespace: percona-pgo
resourceVersion: “346280152”
uid: d5fc665c-84cc-40e0-972b-caeec7be679f
rules:

  • apiGroups:
    • “”
      resources:
    • configmaps
    • endpoints
    • pods
    • pods/exec
    • secrets
    • services
    • persistentvolumeclaims
      verbs:
    • get
    • list
    • watch
    • create
    • patch
    • update
    • delete
    • deletecollection
  • apiGroups:
    • “”
      resources:
    • pods/log
      verbs:
    • get
    • list
    • watch
  • apiGroups:
    • apps
      resources:
    • deployments
    • replicasets
      verbs:
    • get
    • list
    • watch
    • create
    • patch
    • update
    • delete
    • deletecollection
  • apiGroups:
    • batch
      resources:
    • jobs
      verbs:
    • get
    • list
    • watch
    • create
    • patch
    • update
    • delete
    • deletecollection
  • apiGroups:
    • crunchydata.com
      resources:
    • pgclusters
    • pgpolicies
    • pgtasks
    • pgreplicas
      verbs:
    • get
    • list
    • watch
    • create
    • patch
    • update
    • delete
    • deletecollection