Unable to Restore PXC

[Stepa_NF]

Hello support from Percona.

I need to say that i think you guys have been doing good job with kubernetes operator for percona xtradb cluster. Since two weeks ago we started doing a test in our dev environment. We successfully run percona cluster start and running with defined backup job. But, when i tried to start restore backup it does nothing.

Here is definition of backup:

All backup jobs are succeeded.

k -n mysql-ha describe pxc-backup cron-ott-db-fs-pvc-20201125001030-169ad >

cron-ott-db-fs-pvc-20201125001030-169ad.txt

And here is my restore definition:

k -n mysql-ha describe pxc-restore restore-ott-local >

restore-ott-local.txt

The problem is that when i deploy ClusterRestore, I can’t see that restore job actually start. In status field there is nothing. So i have no idea from where start troubleshooting this issue.

In operator i can’t see any related logs when restore is deployed, but there are some other error logs.

pxc-operator-logs.txt

I have one more question, is there possibility to restore individual db from this cluster backup.

I saw there are crds: perconaxtradbclusterbackups, perconaxtradbbackups but just perconaxtradbclusterrestores, not only restore.

Thank you so much in advance,

If you need more information, just ask me.

Best regards,

cron-ott-db-fs-pvc-20201125001030-169ad.txt (1.54 KB)

restore-ott-local.txt (958 Bytes)

pxc-operator-logs.txt (5.83 KB)

[Stepa_NF]

Hi,

I have info about the above problem.

When restarted PXC operator, stuck restore process was just started, and finished successfully.

But still, I don’t know what was the root cause.

[Sergey_Pronin]

@Stepa_NF thank you for submitting it.

I have tried to reproduce the issue and failed.

Have you faced it again? Is there anything special about your setup that might trigger the issue?

[Mahdi_Gerami]

Hi. I also have the same problem! Sometimes the operator doesn’t handle restore custom resources, there are no logs about restore in the operator pod and restore CR has no status as if it’s not being listened or handled. The operator pod still handles Cluster CR but not restore. Restarting the operator pod and redeploying a Restore CR works but the problem happens again and needs restart…

I installed xtradb operator using helm:
chart_version: 2.0.9
mysql_operator_image_version: 8.0.33-2.0.9

[Slava_Sarzhan]

Hi @Mahdi_Gerami we do not support PXC 8.0.33. The latest operator supports 8.0.29. Try to use this version and recheck the restore one more time. The root of the issue can be PXB version in the backup image. It can’t work with PXC 8.0.33.

[Mahdi_Gerami]

Oops! The version I mentioned was for another operator not percona.

For the percona operator helm I’m using:
chart_url: Percona Helm Charts | percona-helm-charts
chart_version: 1.12.2
in the helm values I’m setting these:
operatorImageRepository: percona/percona-xtradb-cluster-operator
imagePullPolicy: IfNotPresent
image: “”

In the PerconaXtraDBCluster custom resource I’m using these images:
for pxc: percona/percona-xtradb-cluster:8.0.29-21.1
backup: percona/percona-xtradb-cluster-operator:1.12.0-pxc8.0-backup