PerconaServerMySQL Fails to Start Up after Restore

MySQL & MariaDB Percona Operator for MySQL
1

Description:

Attempting to migrate a PerconaServerMySQL from one k8s cluster to another using a PerconaServerMySQLRestore. The PerconaServerMySQLBackup is run on an older version of the operator (0.4.0) with the Restore being attempted on a newer version (0.9.1). DBs are set up in the same way using identical credentials.

After the PerconaServerMySQLRestore finishes its operation with a “Succeeded” status the DB pods successfully restart (I can access it with PHPMyAdmin), however the operator cannot contact it anymore and the PerconaServerMySQL resource sits in an error state, logs show:
SELECT command denied to user 'orchestrator'@'172.16.4.20' for table 'replication_group_members'

As a result Orchestrator and HA pods never come back online.

Steps to Reproduce:

  • Perform a backup on PerconaServerMySQL (ver 0.4.0)
  • Restore the backup on PerconaServerMySQL (ver 0.9.1)

Version:

  • Backup run on 0.4.0, Restore run on 0.9.1

Logs:

SELECT command denied to user 'orchestrator'@'172.16.4.20' for table 'replication_group_members'

Expected Result:

  • Operator should be able to reach the DB

Actual Result:

  • Operator unable to reach the DB
2

Alright yeah I should have done this from the beginning.

I rolled the version I’m doing the restore on to the same version of the operator (0.4.0) and it worked as expected now, so I guess there is a limit to how old of a backup you can restore on new versions of the operator somewhere.

3

Hi KSDreams,

Without the error message you are getting I do not know what the problem might be. But sometimes if the amount of versions you are trying to skip is too high some things might not work out of the box and needs some manual intervention.

I suggest you do some intermediate updates instead of patching to latest in one go

Regards

4

If you want the full error log from the operator pod its here:

2025-07-07 14:59:22 ERROR There was an error trying to check group replication information for instance backyard-db-mysql-2.backyard-db-mysql.tools-prod:3306: Error 1142 (42000): SELECT command denied to user 'orchestrator'@'172.16.4.20' for table 'replication_group_members'
github.com/percona/percona-server-mysql-operator/pkg/orchestrator.(*orcResponse).Error
    /go/src/github.com/percona/percona-server-mysql-operator/pkg/orchestrator/client.go:39
github.com/percona/percona-server-mysql-operator/pkg/orchestrator.Discover
    /go/src/github.com/percona/percona-server-mysql-operator/pkg/orchestrator/client.go:234
github.com/percona/percona-server-mysql-operator/pkg/controller/ps.(*PerconaServerMySQLReconciler).reconcileReplication
    /go/src/github.com/percona/percona-server-mysql-operator/pkg/controller/ps/controller.go:861
github.com/percona/percona-server-mysql-operator/pkg/controller/ps.(*PerconaServerMySQLReconciler).doReconcile
    /go/src/github.com/percona/percona-server-mysql-operator/pkg/controller/ps/controller.go:435
github.com/percona/percona-server-mysql-operator/pkg/controller/ps.(*PerconaServerMySQLReconciler).Reconcile
    /go/src/github.com/percona/percona-server-mysql-operator/pkg/controller/ps/controller.go:133
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.1/pkg/internal/controller/controller.go:118
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.1/pkg/internal/controller/controller.go:328
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.1/pkg/internal/controller/controller.go:288
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.1/pkg/internal/controller/controller.go:249
runtime.goexit
    /usr/local/go/src/runtime/asm_amd64.s:1700
failed to discover cluster
github.com/percona/percona-server-mysql-operator/pkg/controller/ps.(*PerconaServerMySQLReconciler).reconcileReplication
    /go/src/github.com/percona/percona-server-mysql-operator/pkg/controller/ps/controller.go:876
github.com/percona/percona-server-mysql-operator/pkg/controller/ps.(*PerconaServerMySQLReconciler).doReconcile
    /go/src/github.com/percona/percona-server-mysql-operator/pkg/controller/ps/controller.go:435
github.com/percona/percona-server-mysql-operator/pkg/controller/ps.(*PerconaServerMySQLReconciler).Reconcile
    /go/src/github.com/percona/percona-server-mysql-operator/pkg/controller/ps/controller.go:133
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.1/pkg/internal/controller/controller.go:118
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.1/pkg/internal/controller/controller.go:328
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.1/pkg/internal/controller/controller.go:288
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.1/pkg/internal/controller/controller.go:249
runtime.goexit
    /usr/local/go/src/runtime/asm_amd64.s:1700
replication
github.com/percona/percona-server-mysql-operator/pkg/controller/ps.(*PerconaServerMySQLReconciler).doReconcile
    /go/src/github.com/percona/percona-server-mysql-operator/pkg/controller/ps/controller.go:436
github.com/percona/percona-server-mysql-operator/pkg/controller/ps.(*PerconaServerMySQLReconciler).Reconcile
    /go/src/github.com/percona/percona-server-mysql-operator/pkg/controller/ps/controller.go:133
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.1/pkg/internal/controller/controller.go:118
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.1/pkg/internal/controller/controller.go:328
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.1/pkg/internal/controller/controller.go:288
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.1/pkg/internal/controller/controller.go:249
runtime.goexit
    /usr/local/go/src/runtime/asm_amd64.s:1700
reconcile
github.com/percona/percona-server-mysql-operator/pkg/controller/ps.(*PerconaServerMySQLReconciler).Reconcile
    /go/src/github.com/percona/percona-server-mysql-operator/pkg/controller/ps/controller.go:134
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.1/pkg/internal/controller/controller.go:118
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.1/pkg/internal/controller/controller.go:328
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.1/pkg/internal/controller/controller.go:288
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.1/pkg/internal/controller/controller.go:249
runtime.goexit
    /usr/local/go/src/runtime/asm_amd64.s:1700

But yeah I’ll ensure the source/destination of the backup/restore are closer to or at the same version.