Crash loop in percona operator Observed a panic in reconciler

PostgreSQL Percona Operator for PostgreSQL
1

Using the operator crVersion: 2.3.1 I was trying to build a new postgres cluster from an existing one by adding a dataSource pointing to an existing cluster. After applying the yaml the operator got stuck in a crash loop with the following stack trace below. There does not seem to be a way to break the crash loop. I even tried deleting the operator and doing a re-install and it immediately resumes the crash loop.

Any suggestions as to how to recover the operator, and what might have caused the error in the
first place?

Thanks.

2024-03-21T19:32:01.194Z INFO Observed a panic in reconciler: runtime error: invalid memory address or nil pointer dereference {"controller": "perconapgcluster", "controllerGroup": "pgv2.percona.com", "controllerKind": "PerconaPGCluster", "PerconaPGCluster": {"name":"rxtest","namespace":"postgres-operator"}, "namespace": "postgres-operator", "name": "jayrxtest", "reconcileID": "0ecffd68-d97a-4d13-af64-9eafd015dd10"}
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
 panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1678ace]
goroutine 459 [running]:
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1()
 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:116 +0x1e5
panic({0x1a233e0?, 0x2ddbe70?})
 /usr/local/go/src/runtime/panic.go:914 +0x21f
github.com/percona/percona-postgresql-operator/pkg/apis/pgv2.percona.com/v2.(*PerconaPGCluster).Default(0xc000cdc380)
 /go/src/github.com/percona/percona-postgresql-operator/pkg/apis/pgv2.percona.com/v2/perconapgcluster_types.go:179 +0x22e
github.com/percona/percona-postgresql-operator/percona/controller/pgcluster.(*PGClusterReconciler).Reconcile(0xc00045ef30, {0x1fcc410?, 0xc000d2b530}, {{{0xc00005ddb8?, 0x5?}, {0xc00083f6f6?, 0xc00044cd48?}}})
 /go/src/github.com/percona/percona-postgresql-operator/percona/controller/pgcluster/controller.go:170 +0x1c5
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x1fcf718?, {0x1fcc410?, 0xc000d2b530?}, {{{0xc00005ddb8?, 0xb?}, {0xc00083f6f6?, 0x0?}}})
 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:119 +0xb7
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc0004e4aa0, {0x1fcc448, 0xc0003a99a0}, {0x1abf5c0?, 0xc000971140?})
 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:316 +0x3cc
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc0004e4aa0, {0x1fcc448, 0xc0003a99a0})
 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:266 +0x1c9
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:227 +0x79
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2 in goroutine 89
 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:223 +0x565
2

@Lobo_Lobo20919 could you please share your custom resource manifest that caused it?
I would love to try to reproduce it.

3

I have faced the same issue and found that it was related to missing spec.proxy.pgBouncer

4

Yup that was it, we have a automated process that builds the cr.yaml and it failed to add the proxy section when testing a restore from one cluster to a new cluster.

I put in a bug report on the percona github.

Thanks for taking a look.

5

@alex2 @Lobo_Lobo20919 thank you for bringing this up.
It will be tracked here: operator crash loop due to nil pointer · Issue #699 · percona/percona-postgresql-operator · GitHub
and in JIRA: [K8SPG-543] - Percona JIRA

Most likely will be fixed in the next release.