Helm installation fails to start patroni cluster

PostgreSQL Percona Operator for PostgreSQL
1

Description:

cluster installed via helm fails to start patroni cluster

Steps to Reproduce:

helm -n test upgrade --create-namespace --install pgsql-operator percona/pg-operator --version 2.2.0 --wait
helm -n test upgrade --create-namespace --install pgsql percona/pg-db --version 2.2.1 --wait

replicas end in endless loop

Logs:

│ database 2023-07-10 13:29:25,739 INFO: Removing data directory: /pgdata/pg15                                                                                                                                                                                                                                                                                                               │
│ database 2023-07-10 13:29:30,710 INFO: Lock owner: pgsql-pg-db-instance1-nttv-0; I am pgsql-pg-db-instance1-g6rp-0                                                                                                                                                                                                                                                                         │
│ database 2023-07-10 13:29:35,712 INFO: trying to bootstrap from leader 'pgsql-pg-db-instance1-nttv-0'                                                                                                                                                                                                                                                                                      │
│ database 2023-07-10 13:29:35,712 ERROR: failed to bootstrap from leader 'pgsql-pg-db-instance1-nttv-0'                                                                                                                                                                                                                                                                                     │
│ database 2023-07-10 13:29:35,712 INFO: Removing data directory: /pgdata/pg15                                                                                                                                                                                                                                                                                                               │
│ database 2023-07-10 13:29:40,734 INFO: Lock owner: pgsql-pg-db-instance1-nttv-0; I am pgsql-pg-db-instance1-g6rp-0                                                                                                                                                                                                                                                                         │
│ database 2023-07-10 13:29:45,741 INFO: trying to bootstrap from leader 'pgsql-pg-db-instance1-nttv-0'                                                                                                                                                                                                                                                                                      │
│ database 2023-07-10 13:29:45,742 ERROR: failed to bootstrap from leader 'pgsql-pg-db-instance1-nttv-0'                                                                                                                                                                                                                                                                                     │
│ database 2023-07-10 13:29:45,742 INFO: Removing data directory: /pgdata/pg15                                                                                                                                                                                                                                                                                                               │
│ database 2023-07-10 13:29:50,713 INFO: Lock owner: pgsql-pg-db-instance1-nttv-0; I am pgsql-pg-db-instance1-g6rp-0                                                                                                                                                                                                                                                                         │
│ database 2023-07-10 13:29:55,719 INFO: trying to bootstrap from leader 'pgsql-pg-db-instance1-nttv-0'                                                                                                                                                                                                                                                                                      │
│ database 2023-07-10 13:29:55,719 ERROR: failed to bootstrap from leader 'pgsql-pg-db-instance1-nttv-0'                                                                                                                                                                                                                                                                                     │
│ database 2023-07-10 13:29:55,719 INFO: Removing data directory: /pgdata/pg15                                                                                                                                                                                                                                                                                                               │
│ database 2023-07-10 13:30:00,691 INFO: Lock owner: pgsql-pg-db-instance1-nttv-0; I am pgsql-pg-db-instance1-g6rp-0                                                                                                                                                                                                                                                                         │
│ database 2023-07-10 13:30:05,698 INFO: trying to bootstrap from leader 'pgsql-pg-db-instance1-nttv-0'                                                                                                                                                                                                                                                                                      │
│ database 2023-07-10 13:30:05,698 ERROR: failed to bootstrap from leader 'pgsql-pg-db-instance1-nttv-0'                                                                                                                                                                                                                                                                                     │
│ database 2023-07-10 13:30:05,698 INFO: Removing data directory: /pgdata/pg15                                                                                                                                                                                                                                                                                                               │
│ database 2023-07-10 13:30:10,720 INFO: Lock owner: pgsql-pg-db-instance1-nttv-0; I am pgsql-pg-db-instance1-g6rp-0                                                                                                                                                                                                                                                                         │
│ database 2023-07-10 13:30:15,726 INFO: trying to bootstrap from leader 'pgsql-pg-db-instance1-nttv-0'                                                                                                                                                                                                                                                                                      │
│ database 2023-07-10 13:30:15,726 ERROR: failed to bootstrap from leader 'pgsql-pg-db-instance1-nttv-0'                                                                                                                                                                                                                                                                                     │
│ database 2023-07-10 13:30:15,726 INFO: Removing data directory: /pgdata/pg15                                                                                                                                                                                                                                                                                                               │
│ database 2023-07-10 13:30:20,698 INFO: Lock owner: pgsql-pg-db-instance1-nttv-0; I am pgsql-pg-db-instance1-g6rp-0                                                                                                                                                                                                                                                                         │
│ database 2023-07-10 13:30:25,705 INFO: trying to bootstrap from leader 'pgsql-pg-db-instance1-nttv-0'                                                                                                                                                                                                                                                                                      │
│ database 2023-07-10 13:30:25,706 ERROR: failed to bootstrap from leader 'pgsql-pg-db-instance1-nttv-0'                                                                                                                                                                                                                                                                                     │
│ database 2023-07-10 13:30:25,706 INFO: Removing data directory: /pgdata/pg15                                                                                                                                                                                                                                                                                                               │
│ database 2023-07-10 13:30:30,728 INFO: Lock owner: pgsql-pg-db-instance1-nttv-0; I am pgsql-pg-db-instance1-g6rp-0                                                                                                                                                                                                                                                                         │
│ database 2023-07-10 13:30:35,735 INFO: trying to bootstrap from leader 'pgsql-pg-db-instance1-nttv-0'                                                                                                                                                                                                                                                                                      │
│ database 2023-07-10 13:30:35,735 ERROR: failed to bootstrap from leader 'pgsql-pg-db-instance1-nttv-0'                                                                                                                                                                                                                                                                                     │
│ database 2023-07-10 13:30:35,735 INFO: Removing data directory: /pgdata/pg15                                                                                                                                                                                                                                                                                                               │
│ database 2023-07-10 13:30:40,706 INFO: Lock owner: pgsql-pg-db-instance1-nttv-0; I am pgsql-pg-db-instance1-g6rp-0                                                                                                                                                                                                                                                                         │
│ database 2023-07-10 13:30:45,711 INFO: trying to bootstrap from leader 'pgsql-pg-db-instance1-nttv-0'                                                                                                                                                                                                                                                                                      │
│ database 2023-07-10 13:30:45,712 ERROR: failed to bootstrap from leader 'pgsql-pg-db-instance1-nttv-0'                                                                                                                                                                                                                                                                                     │
│ database 2023-07-10 13:30:45,712 INFO: Removing data directory: /pgdata/pg15                                                                                                                                                                                                                                                                                                               │
│ database 2023-07-10 13:30:50,733 INFO: Lock owner: pgsql-pg-db-instance1-nttv-0; I am pgsql-pg-db-instance1-g6rp-0                                                                                                                                                                                                                                                                         │
│ database 2023-07-10 13:30:55,738 INFO: trying to bootstrap from leader 'pgsql-pg-db-instance1-nttv-0'                                                                                                                                                                                                                                                                                      │
│ database 2023-07-10 13:30:55,739 ERROR: failed to bootstrap from leader 'pgsql-pg-db-instance1-nttv-0'                                                                                                                                                                                                                                                                                     │
│ database 2023-07-10 13:30:55,739 INFO: Removing data directory: /pgdata/pg15                                                                                                                                                                                                                                                                                                               │
│ database 2023-07-10 13:31:00,710 INFO: Lock owner: pgsql-pg-db-instance1-nttv-0; I am pgsql-pg-db-instance1-g6rp-0                                                                                                                                                                                                                                                                         │
│ database 2023-07-10 13:31:05,717 INFO: trying to bootstrap from leader 'pgsql-pg-db-instance1-nttv-0'                                                                                                                                                                                                                                                                                      │
│ database 2023-07-10 13:31:05,717 ERROR: failed to bootstrap from leader 'pgsql-pg-db-instance1-nttv-0'                                                                                                                                                                                                                                                                                     │
│ database 2023-07-10 13:31:05,717 INFO: Removing data directory: /pgdata/pg15                                                                                                                                                                                                                                                                                                               │
│ database 2023-07-10 13:31:10,738 INFO: Lock owner: pgsql-pg-db-instance1-nttv-0; I am pgsql-pg-db-instance1-g6rp-0                                                                                                                                                                                                                                                                         │
│ database 2023-07-10 13:31:15,745 INFO: trying to bootstrap from leader 'pgsql-pg-db-instance1-nttv-0'                                                                                                                                                                                                                                                                                      │
│ database 2023-07-10 13:31:15,745 ERROR: failed to bootstrap from leader 'pgsql-pg-db-instance1-nttv-0'                                                                                                                                                                                                                                                                                     │
│ database 2023-07-10 13:31:15,745 INFO: Removing data directory: /pgdata/pg15                                                                                                                                                                                                                                                                                                               │
│ database 2023-07-10 13:31:20,716 INFO: Lock owner: pgsql-pg-db-instance1-nttv-0; I am pgsql-pg-db-instance1-g6rp-0                                                                                                                                                                                                                                                                         │
│ database 2023-07-10 13:31:25,721 INFO: trying to bootstrap from leader 'pgsql-pg-db-instance1-nttv-0'                                                                                                                                                                                                                                                                                      │
│ database 2023-07-10 13:31:25,721 ERROR: failed to bootstrap from leader 'pgsql-pg-db-instance1-nttv-0'                                                                                                                                                                                                                                                                                     │
│ database 2023-07-10 13:31:25,721 INFO: Removing data directory: /pgdata/pg15                                                                                                                                                                                                                                                                                                               │
│ database 2023-07-10 13:31:30,692 INFO: Lock owner: pgsql-pg-db-instance1-nttv-0; I am pgsql-pg-db-instance1-g6rp-0                                                                                                                                                                                                                                                                         │
│ database 2023-07-10 13:31:35,698 INFO: trying to bootstrap from leader 'pgsql-pg-db-instance1-nttv-0'                                                                                                                                                                                                                                                                                      │
│ database 2023-07-10 13:31:35,699 ERROR: failed to bootstrap from leader 'pgsql-pg-db-instance1-nttv-0'                                                                                                                                                                                                                                                                                     │
│ database 2023-07-10 13:31:35,699 INFO: Removing data directory: /pgdata/pg15                                                                                                                                                                                                                                                                                                               │
│ database 2023-07-10 13:31:40,720 INFO: Lock owner: pgsql-pg-db-instance1-nttv-0; I am pgsql-pg-db-instance1-g6rp-0                                                                                                                                                                                                                                                                         │
│ database 2023-07-10 13:31:45,727 INFO: trying to bootstrap from leader 'pgsql-pg-db-instance1-nttv-0'                                                                                                                                                                                                                                                                                      │
│ database 2023-07-10 13:31:45,728 ERROR: failed to bootstrap from leader 'pgsql-pg-db-instance1-nttv-0'                                                                                                                                                                                                                                                                                     │
│ database 2023-07-10 13:31:45,728 INFO: Removing data directory: /pgdata/pg15                                                                                                                                                                                                                                                                                                               │
│ database 2023-07-10 13:31:50,699 INFO: Lock owner: pgsql-pg-db-instance1-nttv-0; I am pgsql-pg-db-instance1-g6rp-0                                                                                                                                                                                                                                                                         │
│ database 2023-07-10 13:31:55,705 INFO: trying to bootstrap from leader 'pgsql-pg-db-instance1-nttv-0'                                                                                                                                                                                                                                                                                      │
│ database 2023-07-10 13:31:55,705 ERROR: failed to bootstrap from leader 'pgsql-pg-db-instance1-nttv-0'                                                                                                                                                                                                                                                                                     │
│ database 2023-07-10 13:31:55,706 INFO: Removing data directory: /pgdata/pg15

Environment: GKE private cluster, k8s v1.26.5-gke.1200

2

Hello @rzc96133 ,

I have the same GKE version: GitVersion:“v1.26.5-gke.2100”
I executed the same steps as you did and my cluster is healthy.
Is there anything else that you changed?

3

Hello @Sergey_Pronin

nothing else, is there anything i can gather to debug?

4

Unfortunatelly nothing that I can think of.
We can have a call and do a debugging session if you are open to it.