# "Observed a panic in reconciler" on the backup restore request \[perconaxtradbclusterrestore.pxc.percona.com\] **URL:** \ **Category:** Percona Operator for MySQL\ **Tags:** mysql, percona, kubernetes\ **Created:** [May 9, 2024, 5:50pm UTC](https://forums.percona.com/t/observed-a-panic-in-reconciler-on-the-backup-restore-request-perconaxtradbclusterrestore-pxc-percona-com/30192 "2024-05-09T17:50:43Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![helgie](https://avatars.discourse-cdn.com/v4/letter/h/76d3ee/32.png) [@helgie](https://forums.percona.com/u/helgie)\ **Post date:** [May 9, 2024, 5:50pm UTC](https://forums.percona.com/t/observed-a-panic-in-reconciler-on-the-backup-restore-request-perconaxtradbclusterrestore-pxc-percona-com/30192/1 "2024-05-09T17:50:43Z")
Hello, I am following some instructions from [this document](https://docs.percona.com/percona-operator-for-mysql/pxc/backups-move-from-external-db.html) in order to restore my PXC cluster from my backup located on S3. As my first step, I have created a secret and installed PMM into the freshly created namespace. ```auto helm install pmm --namespace percona --set secret.create=false --set secret.name=pmm-secret percona/pmm ``` Then I have installed Percona Operator for MySQL into the same NS: ```auto helm install percona-operator percona/pxc-operator --namespace percona ``` And the Percona XtraDB Cluster itself alongside: ```auto --set haproxy.nodeSelector.cluster=percona-cluster-node \ --set pxc.nodeSelector.cluster=percona-cluster-node \ --set pxc.resources.requests.cpu=2500m \ --set pxc.resources.requests.memory=6Gi \ --set pxc.resources.limits.cpu=2500m \ --set pxc.resources.limits.memory=6Gi \ --set pxc.persistence.enabled=true \ --set pxc.persistence.size=15Gi \ --set pxc.persistence.storageClass=hcloud-volumes \ --set pmm.enabled=true \ --set pmm.resources.requests.cpu=200m \ --set pmm.resources.requests.memory=150M \ --set pmm.resources.limits.cpu=200m \ --set pmm.resources.limits.memory=150M \ --set haproxy.resources.requests.cpu=300m \ --set haproxy.resources.requests.memory=512M \ --set haproxy.resources.limits.cpu=300m \ --set haproxy.resources.limits.memory=512M \ --set logcollector.resources.requests.cpu=100m \ --set logcollector.resources.requests.memory=100M \ --set backup.enabled=false \ --set secrets.passwords.root=$(openssl rand -base64 32 | tr -d '\n') \ --set secrets.passwords.pmmserverkey= ************************************** \ --set secrets.passwords.xtrabackup=$(openssl rand -base64 32 | tr -d '\n') \ --set secrets.passwords.replication=$(openssl rand -base64 32 | tr -d '\n') \ --set secrets.passwords.proxyadmin=$(openssl rand -base64 32 | tr -d '\n') \ --set secrets.passwords.operator=$(openssl rand -base64 32 | tr -d '\n') \ --set secrets.passwords.monitor=$(openssl rand -base64 32 | tr -d '\n') \ --set secrets.passwords.clustercheck=$(openssl rand -base64 32 | tr -d '\n') ``` At that point the whole setup seemed healthy and functional to me as I was able to login into the PMM Dashboard and to observe some cluster data there, I could spot no restarting pods or whatever on my k8s-dashboard as well. ```auto root@master-001:~/k8s/pxc# kubectl get all -n percona NAME READY STATUS RESTARTS AGE pod/percona-cluster-pxc-d-haproxy-0 3/3 Running 0 179m pod/percona-cluster-pxc-d-haproxy-1 3/3 Running 0 179m pod/percona-cluster-pxc-d-haproxy-2 3/3 Running 0 3h pod/percona-cluster-pxc-d-pxc-0 4/4 Running 0 3h pod/percona-cluster-pxc-d-pxc-1 4/4 Running 0 3h1m pod/percona-cluster-pxc-d-pxc-2 4/4 Running 0 3h2m pod/percona-operator-pxc-operator-687f56d59d-4hkqj 1/1 Running 0 18h pod/pmm-0 1/1 Running 0 46h NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/monitoring-service NodePort 10.xxx.xxx.211 443:31958/TCP,80:32217/TCP 46h service/percona-cluster-pxc-d-haproxy ClusterIP 10.xxx.xxx.59 3306/TCP,3309/TCP,33062/TCP,33060/TCP 3h4m service/percona-cluster-pxc-d-haproxy-replicas ClusterIP 10.xxx.xxx.92 3306/TCP 3h4m service/percona-cluster-pxc-d-pxc ClusterIP None 3306/TCP,33062/TCP,33060/TCP 3h4m service/percona-cluster-pxc-d-pxc-unready ClusterIP None 3306/TCP,33062/TCP,33060/TCP 3h4m service/percona-xtradb-cluster-operator ClusterIP 10.xxx.xxx.172 443/TCP 18h NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/percona-operator-pxc-operator 1/1 1 1 18h NAME DESIRED CURRENT READY AGE replicaset.apps/percona-operator-pxc-operator-687f56d59d 1 1 1 18h NAME READY AGE statefulset.apps/percona-cluster-pxc-d-haproxy 3/3 3h4m statefulset.apps/percona-cluster-pxc-d-pxc 3/3 3h4m statefulset.apps/pmm 1/1 46h ``` As for my source environment, it is a regular Ubuntu 22.04 Server with mysql-server installed there. I have installed Xtrabackup 8 and made a succesful backup to my AWS S3 bucket. ```auto 370 wget https://repo.percona.com/apt/percona-release_latest.$(lsb_release -sc)_all.deb 371 dpkg -i percona-release_latest.$(lsb_release -sc)_all.deb 372 percona-release setup ps80 373 sudo apt update 374 sudo apt install percona-xtrabackup-80 375 sudo apt install lz4 376 xtrabackup 395 xtrabackup --backup --stream=xbstream --extra-lsndir=/tmp --target-dir=/tmp | xbcloud put --storage=s3 --s3-endpoint='s3.amazonaws.com' --s3-region='eu-central-1' --s3-access-key=$AWS_ACCESS_KEY_ID --s3-secret-key=$AWS_SECRET_ACCESS_KEY --s3-bucket=' **************** -percona-db-backups' --parallel=10 $(date -I)-full_backup ``` This is what I can see inside my S3 bucket ```auto % aws s3 ls **************** -percona-db-backups/2024-05-08-full_backup/ PRE app/ PRE mysql/ PRE performance_schema/ PRE sys/ 2024-05-09 00:17:50 496 backup-my.cnf.00000000000000000000 2024-05-09 00:17:50 27 backup-my.cnf.00000000000000000001 2024-05-09 00:17:50 244 binlog.000150.00000000000000000000 2024-05-09 00:17:50 27 binlog.000150.00000000000000000001 2024-05-09 00:17:50 62 binlog.index.00000000000000000000 2024-05-09 00:17:50 26 binlog.index.00000000000000000001 2024-05-09 00:17:50 759599 ib_buffer_pool.00000000000000000000 2024-05-09 00:17:50 28 ib_buffer_pool.00000000000000000001 2024-05-09 00:17:23 10485801 ibdata1.00000000000000000000 2024-05-09 00:17:23 10485801 ibdata1.00000000000000000001 2024-05-09 00:17:23 10485801 ibdata1.00000000000000000002 2024-05-09 00:17:23 10485801 ibdata1.00000000000000000003 2024-05-09 00:17:23 10485801 ibdata1.00000000000000000004 2024-05-09 00:17:23 10485801 ibdata1.00000000000000000005 2024-05-09 00:17:23 10485801 ibdata1.00000000000000000006 2024-05-09 00:17:23 6291497 ibdata1.00000000000000000007 2024-05-09 00:17:23 21 ibdata1.00000000000000000008 2024-05-09 00:17:49 10485803 mysql.ibd.00000000000000000000 2024-05-09 00:17:49 10485803 mysql.ibd.00000000000000000001 2024-05-09 00:17:49 10485803 mysql.ibd.00000000000000000002 2024-05-09 00:17:49 10485803 mysql.ibd.00000000000000000003 2024-05-09 00:17:49 4194347 mysql.ibd.00000000000000000004 2024-05-09 00:17:49 23 mysql.ibd.00000000000000000005 2024-05-09 00:17:50 10485802 undo_001.00000000000000000000 2024-05-09 00:17:50 6291498 undo_001.00000000000000000001 2024-05-09 00:17:50 22 undo_001.00000000000000000002 2024-05-09 00:17:50 10485802 undo_002.00000000000000000000 2024-05-09 00:17:50 6291498 undo_002.00000000000000000001 2024-05-09 00:17:50 22 undo_002.00000000000000000002 2024-05-09 00:17:50 116 xtrabackup_binlog_info.00000000000000000000 2024-05-09 00:17:50 36 xtrabackup_binlog_info.00000000000000000001 2024-05-09 00:17:50 199 xtrabackup_checkpoints.00000000000000000000 2024-05-09 00:17:50 36 xtrabackup_checkpoints.00000000000000000001 2024-05-09 00:17:51 623 xtrabackup_info.00000000000000000000 2024-05-09 00:17:51 29 xtrabackup_info.00000000000000000001 2024-05-09 00:17:50 3124 xtrabackup_logfile.00000000000000000000 2024-05-09 00:17:50 32 xtrabackup_logfile.00000000000000000001 2024-05-09 00:17:51 95 xtrabackup_tablespaces.00000000000000000000 2024-05-09 00:17:51 36 xtrabackup_tablespaces.00000000000000000001 ``` On my target environment, I have created a secret to store my AWS credentials, which are identical to the ones mentioned above: ```auto root@master-001:~/k8s/pxc# kubectl get secret **************** -percona-db-backups -n percona -o yaml apiVersion: v1 data: AWS_ACCESS_KEY_ID: ************************ AWS_SECRET_ACCESS_KEY: ******************** kind: Secret metadata: creationTimestamp: "2024-05-09T14:55:28Z" name: **************** -percona-db-backups namespace: percona resourceVersion: "5024887" uid: 2b468225-2d6f-490f-899d-67725b4f0d23 type: Opaque ``` I have created the following k8s-manifest to request a restore: ```auto root@master-001:~/k8s/pxc# cat initial-restore.yaml apiVersion: pxc.percona.com/v1 kind: PerconaXtraDBClusterRestore metadata: name: initial-restore spec: pxcCluster: percona-cluster-pxc-d backupSource: destination: s3:// **************** -percona-db-backups/2024-05-08-full_backup s3: credentialsSecret: **************** -percona-db-backups endpointUrl: https://s3.eu-central-1.amazonaws.com region: eu-central-1 pitr: type: latest backupSource: s3: bucket: " **************** -percona-db-backups" credentialsSecret: **************** -percona-db-backups endpointUrl: https://s3.eu-central-1.amazonaws.com region: eu-central-1 ``` And applying it also seemed to be fine: ```auto root@master-001:~/k8s/pxc# kubectl apply -f initial-restore.yaml -n percona perconaxtradbclusterrestore.pxc.percona.com/initial-restore created root@master-001:~/k8s/pxc# kubectl describe pxc-restore/initial-restore -n percona Name: initial-restore Namespace: percona Labels: Annotations: API Version: pxc.percona.com/v1 Kind: PerconaXtraDBClusterRestore Metadata: Creation Timestamp: 2024-05-09T17:42:18Z Generation: 1 Resource Version: 5052630 UID: 6917c770-e75d-45f8-b4d0-60fda2f9374b Spec: Backup Source: Destination: s3:// **************** -percona-db-backups/2024-05-08-full_backup s3: Credentials Secret: **************** -percona-db-backups Endpoint URL: https://s3.eu-central-1.amazonaws.com Region: eu-central-1 Pitr: Backup Source: s3: Bucket: **************** -percona-db-backups Credentials Secret: **************** -percona-db-backups Endpoint URL: https://s3.eu-central-1.amazonaws.com Region: eu-central-1 Type: latest Pxc Cluster: percona-cluster-pxc-d Status: Comments: You can view xtrabackup log: $ kubectl logs job/restore-job-initial-restore-percona-cluster-pxc-d If everything is fine, you can cleanup the job: $ kubectl delete pxc-restore/initial-restore Completed: 2024-05-09T17:42:18Z State: Succeeded Events: ``` I have noticed that no job was created and decided to trace my percona-operator logs to find the reason. ```auto root@master-001:~# kubectl logs pod/percona-operator-pxc-operator-687f56d59d-4hkqj -n percona 2024-05-09T16:34:18.148Z INFO setup Runs on {"platform": "kubernetes", "version": "v1.27.1"} 2024-05-09T16:34:18.149Z INFO setup Manager starting up {"gitCommit": "c85a021f2a21441500b02a2c0b3d17e8a8b25996", "gitBranch": "release-1-14-0", "buildTime": "2024-03-01T09:01:29Z", "goVersion": "go1.21.7", "os": "linux", "arch": "amd64"} 2024-05-09T16:34:18.149Z INFO setup Registering Components. 2024-05-09T16:34:19.300Z INFO controller-runtime.webhook Registering webhook {"path": "/validate-percona-xtradbcluster"} 2024-05-09T16:34:19.300Z INFO setup Starting the Cmd. 2024-05-09T16:34:19.300Z INFO controller-runtime.metrics Starting metrics server 2024-05-09T16:34:19.300Z INFO controller-runtime.metrics Serving metrics server {"bindAddress": ":8080", "secure": false} 2024-05-09T16:34:19.300Z INFO starting server {"kind": "health probe", "addr": "[::]:8081"} 2024-05-09T16:34:19.301Z INFO controller-runtime.webhook Starting webhook server 2024-05-09T16:34:19.301Z INFO controller-runtime.certwatcher Updated current TLS certificate 2024-05-09T16:34:19.301Z INFO controller-runtime.webhook Serving webhook server {"host": "", "port": 9443} 2024-05-09T16:34:19.301Z INFO controller-runtime.certwatcher Starting certificate watcher 2024-05-09T16:34:19.302Z INFO attempting to acquire leader lease percona/08db1feb.percona.com... 2024-05-09T16:34:35.256Z INFO successfully acquired lease percona/08db1feb.percona.com 2024-05-09T16:34:35.256Z INFO Starting EventSource {"controller": "pxc-controller", "source": "kind source: *v1.PerconaXtraDBCluster"} 2024-05-09T16:34:35.256Z INFO Starting Controller {"controller": "pxc-controller"} 2024-05-09T16:34:35.256Z INFO Starting EventSource {"controller": "pxcrestore-controller", "source": "kind source: *v1.PerconaXtraDBClusterRestore"} 2024-05-09T16:34:35.256Z INFO Starting Controller {"controller": "pxcrestore-controller"} 2024-05-09T16:34:35.257Z INFO Starting EventSource {"controller": "pxcbackup-controller", "source": "kind source: *v1.PerconaXtraDBClusterBackup"} 2024-05-09T16:34:35.257Z INFO Starting Controller {"controller": "pxcbackup-controller"} 2024-05-09T16:34:35.376Z INFO Starting workers {"controller": "pxcbackup-controller", "worker count": 1} 2024-05-09T16:34:35.405Z INFO Starting workers {"controller": "pxc-controller", "worker count": 1} 2024-05-09T16:34:35.407Z INFO Starting workers {"controller": "pxcrestore-controller", "worker count": 1} 2024-05-09T16:34:37.130Z INFO add new job {"controller": "pxc-controller", "namespace": "percona", "name": "percona-cluster-pxc-d", "reconcileID": "d4b88387-4e1c-42a8-899a-026b61667557", "name": "ensure-version/percona/percona-cluster-pxc-d", "schedule": "0 4 * * *"} 2024-05-09T16:34:37.130Z INFO add new job {"controller": "pxc-controller", "namespace": "percona", "name": "percona-cluster-pxc-d", "reconcileID": "d4b88387-4e1c-42a8-899a-026b61667557", "name": "telemetry/percona/percona-cluster-pxc-d", "schedule": "47 * * * *"} root@master-001:~# kubectl logs pod/percona-operator-pxc-operator-687f56d59d-4hkqj -n percona --follow 2024-05-09T16:34:18.148Z INFO setup Runs on {"platform": "kubernetes", "version": "v1.27.1"} 2024-05-09T16:34:18.149Z INFO setup Manager starting up {"gitCommit": "c85a021f2a21441500b02a2c0b3d17e8a8b25996", "gitBranch": "release-1-14-0", "buildTime": "2024-03-01T09:01:29Z", "goVersion": "go1.21.7", "os": "linux", "arch": "amd64"} 2024-05-09T16:34:18.149Z INFO setup Registering Components. 2024-05-09T16:34:19.300Z INFO controller-runtime.webhook Registering webhook {"path": "/validate-percona-xtradbcluster"} 2024-05-09T16:34:19.300Z INFO setup Starting the Cmd. 2024-05-09T16:34:19.300Z INFO controller-runtime.metrics Starting metrics server 2024-05-09T16:34:19.300Z INFO controller-runtime.metrics Serving metrics server {"bindAddress": ":8080", "secure": false} 2024-05-09T16:34:19.300Z INFO starting server {"kind": "health probe", "addr": "[::]:8081"} 2024-05-09T16:34:19.301Z INFO controller-runtime.webhook Starting webhook server 2024-05-09T16:34:19.301Z INFO controller-runtime.certwatcher Updated current TLS certificate 2024-05-09T16:34:19.301Z INFO controller-runtime.webhook Serving webhook server {"host": "", "port": 9443} 2024-05-09T16:34:19.301Z INFO controller-runtime.certwatcher Starting certificate watcher 2024-05-09T16:34:19.302Z INFO attempting to acquire leader lease percona/08db1feb.percona.com... 2024-05-09T16:34:35.256Z INFO successfully acquired lease percona/08db1feb.percona.com 2024-05-09T16:34:35.256Z INFO Starting EventSource {"controller": "pxc-controller", "source": "kind source: *v1.PerconaXtraDBCluster"} 2024-05-09T16:34:35.256Z INFO Starting Controller {"controller": "pxc-controller"} 2024-05-09T16:34:35.256Z INFO Starting EventSource {"controller": "pxcrestore-controller", "source": "kind source: *v1.PerconaXtraDBClusterRestore"} 2024-05-09T16:34:35.256Z INFO Starting Controller {"controller": "pxcrestore-controller"} 2024-05-09T16:34:35.257Z INFO Starting EventSource {"controller": "pxcbackup-controller", "source": "kind source: *v1.PerconaXtraDBClusterBackup"} 2024-05-09T16:34:35.257Z INFO Starting Controller {"controller": "pxcbackup-controller"} 2024-05-09T16:34:35.376Z INFO Starting workers {"controller": "pxcbackup-controller", "worker count": 1} 2024-05-09T16:34:35.405Z INFO Starting workers {"controller": "pxc-controller", "worker count": 1} 2024-05-09T16:34:35.407Z INFO Starting workers {"controller": "pxcrestore-controller", "worker count": 1} 2024-05-09T16:34:37.130Z INFO add new job {"controller": "pxc-controller", "namespace": "percona", "name": "percona-cluster-pxc-d", "reconcileID": "d4b88387-4e1c-42a8-899a-026b61667557", "name": "ensure-version/percona/percona-cluster-pxc-d", "schedule": "0 4 * * *"} 2024-05-09T16:34:37.130Z INFO add new job {"controller": "pxc-controller", "namespace": "percona", "name": "percona-cluster-pxc-d", "reconcileID": "d4b88387-4e1c-42a8-899a-026b61667557", "name": "telemetry/percona/percona-cluster-pxc-d", "schedule": "47 * * * *"} 2024-05-09T16:36:54.361Z INFO backup restore request {"controller": "pxcrestore-controller", "namespace": "percona", "name": "initial-restore", "reconcileID": "30883363-e4e0-4dd6-93f1-6f8c00c85be1"} 2024-05-09T16:36:54.392Z INFO Observed a panic in reconciler: runtime error: invalid memory address or nil pointer dereference {"controller": "pxcrestore-controller", "namespace": "percona", "name": "initial-restore", "reconcileID": "30883363-e4e0-4dd6-93f1-6f8c00c85be1"} panic: runtime error: invalid memory address or nil pointer dereference [recovered] panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x58 pc=0x1634635] goroutine 197 [running]: sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1() /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/controller/controller.go:116 +0x1e5 panic({0x1aa5fe0?, 0x2e707b0?}) /usr/local/go/src/runtime/panic.go:914 +0x21f github.com/percona/percona-xtradb-cluster-operator/pkg/apis/pxc/v1.(*PXCBackupStatus).GetStorageType(...) /go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/apis/pxc/v1/pxc_backup_types.go:140 github.com/percona/percona-xtradb-cluster-operator/pkg/pxc/backup.RestoreJob(0xc000e901a0, 0xc000353200, 0xc000ad0a00, {0xc000c4ca00, 0x45}, 0x0) /go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/pxc/backup/restore.go:140 +0x75 github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcrestore.(*s3).Job(0xc00065e5d0?) /go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcrestore/restorer.go:38 +0x32 github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcrestore.(*ReconcilePerconaXtraDBClusterRestore).validate(0x1a8a960?, {0x204fa08, 0xc000f3ef90}, 0xc000e901a0, 0x204f998?, 0xc000ad0a00?) /go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcrestore/restore.go:80 +0x4b github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcrestore.(*ReconcilePerconaXtraDBClusterRestore).Reconcile(0xc00065e5d0, {0x204fa08, 0xc000f3ef90}, {{{0xc000992606?, 0x5?}, {0xc000992610?, 0xc000ec6d08?}}}) /go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcrestore/controller.go:190 +0xf14 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x2053448?, {0x204fa08?, 0xc000f3ef90?}, {{{0xc000992606?, 0xb?}, {0xc000992610?, 0x0?}}}) /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/controller/controller.go:119 +0xb7 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc0003f20a0, {0x204fa40, 0xc000607a90}, {0x1b4eb40?, 0xc000bf4560?}) /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/controller/controller.go:316 +0x3cc sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc0003f20a0, {0x204fa40, 0xc000607a90}) /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/controller/controller.go:266 +0x1af sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2() /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/controller/controller.go:227 +0x79 created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2 in goroutine 130 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/controller/controller.go:223 +0x565 ``` Could you please kindly tell if I am doing something wrong or is it a defect? TIA ---
**Author:** ![Frederic\_Coelho1](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/frederic_coelho1/32/12460_2.png) [@Frederic\_Coelho1](https://forums.percona.com/u/Frederic_Coelho1)\ **Post date:** [June 5, 2024, 11:20am UTC](https://forums.percona.com/t/observed-a-panic-in-reconciler-on-the-backup-restore-request-perconaxtradbclusterrestore-pxc-percona-com/30192/2 "2024-06-05T11:20:04Z")
you must activate the backup with a storage otherwise the operator panic ---
**Author:** ![NeerajSharma](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/neerajsharma/32/17520_2.png) [@NeerajSharma](https://forums.percona.com/u/NeerajSharma)\ **Post date:** [August 4, 2024, 7:44pm UTC](https://forums.percona.com/t/observed-a-panic-in-reconciler-on-the-backup-restore-request-perconaxtradbclusterrestore-pxc-percona-com/30192/3 "2024-08-04T19:44:01Z")
Hi helgie, Have you find the solution as I am also facing the same issue ---
**Author:** ![Inel\_Pandzic](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/inel_pandzic/32/23166_2.png) [@Inel\_Pandzic](https://forums.percona.com/u/Inel_Pandzic)\ **Post date:** [September 5, 2024, 1:29pm UTC](https://forums.percona.com/t/observed-a-panic-in-reconciler-on-the-backup-restore-request-perconaxtradbclusterrestore-pxc-percona-com/30192/4 "2024-09-05T13:29:48Z")
Hello, Thanks for reporting the issue. We found the problem and have a ticket to fix it in the next release: [https://perconadev.atlassian.net/browse/K8SPXC-1458](https://perconadev.atlassian.net/browse/K8SPXC-1458) A workaround right now is what @Frederic_Coelho1 suggested in his reply. Basically in your target cluster set backup and storage fields in the CR. ---
**Author:** ![NeerajSharma](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/neerajsharma/32/17520_2.png) [@NeerajSharma](https://forums.percona.com/u/NeerajSharma)\ **Post date:** [September 8, 2024, 8:01pm UTC](https://forums.percona.com/t/observed-a-panic-in-reconciler-on-the-backup-restore-request-perconaxtradbclusterrestore-pxc-percona-com/30192/5 "2024-09-08T20:01:22Z")
Hi @Inel_Pandzic Thanks for helping me out. I have enabled the backup storage of target cluster and then tried restoring it and this time I am not getting Operator panic runtime error. But after restoring the cluster pxc pod not getting ready and showing error : Readiness probe failed: ERROR 1045 (28000): Access denied for user ‘monitor’@‘mysql-restore-pxc-0.mysql-restore-pxc.mysql-restore.svc.cluster.’ (using password: YES) - [[‘’ == \P\r\i\m\a\r\y]] - exit 1 Steps to produce: 1. I started the target cluster is ‘mysql-restore’ namespace with 2 pxc pods and 1 proxysql pod. Config: [pxc-restore.txt](https://forums.percona.com/uploads/short-url/im3shBLOpzozGiG19SI4VhIlZOt.txt) (5.0 KB) ![Screenshot from 2024-09-09 01-22-08](https://us1.discourse-cdn.com/flex019/uploads/percona1/original/3X/6/e/6ee1b479c671b30f3398f0b526dec5b6bedbcc88.png) I exec into mysql-restore-pxc-0 and tried connecting to server using both root and monitor user and able to access the db 2. Then I applied the restore manifest file that has backup and pitr S3 source (S3 credential secrets is already appllied) Config: [restore.txt](https://forums.percona.com/uploads/short-url/4mXMBmPDLQ9wPNFIkEpATQiSNaR.txt) (656 Bytes) 3. First it stopped the running cluster. Started replication job that completed successfully ![Screenshot from 2024-09-09 01-27-47](https://us1.discourse-cdn.com/flex019/uploads/percona1/original/3X/8/6/8684c90b2463b02c6730d52c6e0902e3c901b578.png) 4. Once that cluster restarted. Proxysql pod gets in ready state with 3/3 (all 3 container started successfully). But for pxc 2/3 (pxc pod is failing due to readiness probe failure) ![Screenshot from 2024-09-09 01-29-18](https://us1.discourse-cdn.com/flex019/uploads/percona1/original/3X/5/f/5fbb3f12ff5bbcd729fb5167fe2c4c9fb06dcfec.png) In operator no logs generated. Can you please help me in resolving this issue ---
**Author:** ![Slava\_Sarzhan](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/slava_sarzhan/32/3196_2.png) [@Slava\_Sarzhan](https://forums.percona.com/u/Slava_Sarzhan)\ **Post date:** [September 26, 2024, 5:25pm UTC](https://forums.percona.com/t/observed-a-panic-in-reconciler-on-the-backup-restore-request-perconaxtradbclusterrestore-pxc-percona-com/30192/6 "2024-09-26T17:25:41Z")
@NeerajSharma, do you have the same secrets in both your clusters? P.S. the following issue was merged [[K8SPXC-1458] - Percona JIRA](https://perconadev.atlassian.net/browse/K8SPXC-1458) and will be included in the next PXCO release. ---
**Author:** ![Frederic\_Coelho1](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/frederic_coelho1/32/12460_2.png) [@Frederic\_Coelho1](https://forums.percona.com/u/Frederic_Coelho1)\ **Post date:** [December 17, 2024, 11:58am UTC](https://forums.percona.com/t/observed-a-panic-in-reconciler-on-the-backup-restore-request-perconaxtradbclusterrestore-pxc-percona-com/30192/7 "2024-12-17T11:58:58Z")
wonderful, thank you ---
**Author:** ![Slava\_Sarzhan](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/slava_sarzhan/32/3196_2.png) [@Slava\_Sarzhan](https://forums.percona.com/u/Slava_Sarzhan)\ **Post date:** [January 13, 2025, 4:15pm UTC](https://forums.percona.com/t/observed-a-panic-in-reconciler-on-the-backup-restore-request-perconaxtradbclusterrestore-pxc-percona-com/30192/8 "2025-01-13T16:15:56Z")
@NeerajSharma @Frederic_Coelho1 the issue was fixed. PXCO 1.16 is available for deployment.