Trying to figure out root cause for crash. during upgrade from 1.2 to 1.3. deployment in GCP
Unable to upgrade it just hangs ** percona-postgresql-operator v2.3.1 will be upgraded to v2.4.1
% everestctl upgrade
🚀 You're running the latest version of Everest
% kubectl get csv -n everest-system
NAME DISPLAY VERSION REPLACES PHASE
everest-operator.v1.2.0 Everest operator 1.2.0 everest-operator.v1.1.1 Installing
everest-operator.v1.3.0 Everest operator 1.3.0 everest-operator.v1.2.0 Pending
% kubectl get po -n everest-system
NAME READY STATUS RESTARTS AGE
everest-operator-controller-manager-5c88b98665-4xpzp 1/2 CrashLoopBackOff 5711 (56s ago) 21d
everest-operator-controller-manager-74f4d7bbdb-hct7q 1/2 CrashLoopBackOff 2454 (3m20s ago) 8d
percona-everest-5db6cf7d78-xv9b6 1/1 Running 0 8d
pod logs
**% kubectl logs -n everest-system everest-operator-controller-manager-5c88b98665-4xpzp**
...
I0103 15:08:37.106787 1 leaderelection.go:260] successfully acquired lease everest-system/9094838c.percona.com
2025-01-03T15:08:37Z DEBUG events everest-operator-controller-manager-5c88b98665-4xpzp_b1e8974d-070c-4e70-a99a-b80512286a42 became leader {"type": "Normal", "object": {"kind":"Lease","namespace":"everest-system","name":"9094838c.percona.com","uid":"0a204a40-c63d-4078-90e2-6e754166c96c","apiVersion":"coordination.k8s.io/v1","resourceVersion":"310948517"}, "reason": "LeaderElection"}
...
st": {"name":"sql-btest","namespace":"everest-dev"}}
2025-01-03T15:08:37Z INFO Observed a panic in reconciler: assignment to entry in nil map {"controller": "databasecluster", "controllerGroup": "everest.percona.com", "controllerKind": "DatabaseCluster", "DatabaseCluster": {"name":"sql-btest","namespace":"everest-dev"}, "namespace": "everest-dev", "name": "sql-btest", "reconcileID": "a0f2624d-7d91-417f-a0a4-931245dd8577"}
panic: assignment to entry in nil map [recovered]
panic: assignment to entry in nil map
goroutine 1085 [running]:
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1()
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:111 +0x1e5
panic({0x1f207a0?, 0x294b810?})
/usr/local/go/src/runtime/panic.go:770 +0x132
github.com/percona/everest-operator/controllers/providers/pxc.(*applier).Engine(0xc0007c6ff0)
/workspace/controllers/providers/pxc/applier.go:92 +0x585
github.com/percona/everest-operator/controllers.(*DatabaseClusterReconciler).reconcileDB.func1()
/workspace/controllers/databasecluster_controller.go:153 +0x7e
sigs.k8s.io/controller-runtime/pkg/controller/controllerutil.mutate(0xc001356508?, {{0xc0015d2720?, 0x0?}, {0xc0015d2710?, 0x2972e68?}}, {0x298b8e0, 0xc001356508})
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/controller/controllerutil/controllerutil.go:442 +0x49
sigs.k8s.io/controller-runtime/pkg/controller/controllerutil.CreateOrUpdate({0x2972e68, 0xc000db7c80}, {0x297df60, 0xc0005bd4d0}, {0x298b8e0, 0xc001356508}, 0xc000ab3af8)
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/controller/controllerutil/controllerutil.go:308 +0x23d
github.com/percona/everest-operator/controllers.(*DatabaseClusterReconciler).reconcileDB(0xc0005a2200, {0x2972e68, 0xc000db7c80}, 0xc001602008, {0x298c500, 0xc00051a8c0})
/workspace/controllers/databasecluster_controller.go:170 +0x1e6
github.com/percona/everest-operator/controllers.(*DatabaseClusterReconciler).Reconcile(0xc0005a2200, {0x2972e68, 0xc000db7c80}, {{{0xc000db24a0, 0xb}, {0xc000db2490, 0x9}}})
/workspace/controllers/databasecluster_controller.go:270 +0x509
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x2977b08?, {0x2972e68?, 0xc000db7c80?}, {{{0xc000db24a0?, 0xb?}, {0xc000db2490?, 0x0?}}})
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:114 +0xb7
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc000a24000, {0x2972ea0, 0xc0008f1860}, {0x1febce0, 0xc000674f60})
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:311 +0x3bc
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc000a24000, {0x2972ea0, 0xc0008f1860})
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:261 +0x1be
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:222 +0x79
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2 in goroutine 292
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:218 +0x486
**% kubectl logs -n everest-system everest-operator-controller-manager-74f4d7bbdb-hct7q**
2025-01-03T15:12:11Z ERROR setup MONITORING_NAMESPACE must be set {"error": "failed to get the monitoring namespace"}
main.main
/workspace/main.go:89
runtime.main
/usr/local/go/src/runtime/proc.go:271