Percona XtraDB operator with single node detects itself as peer

Joshua_Sierles · July 11, 2024, 1:31pm

Hello,

I’m looking into the Percona XtraDB k8s Operator bootstrap behavior when running a single node. When booting a single node, it immediately goes into ‘crash mode’.

In theory, a first time single node should not setup replication, and should bootstrap, correct?

In the pxc node entrypoint, this section always runs, and the node detects itself as a peer, which switches on replication. This sets CLUSTER_JOIN, which instructs the node not to bootstrap here.

Perhaps I’m misunderstanding something. Thanks.

Pep_Pla · July 11, 2024, 4:11pm

Hi Joshua,

The Percona Operator for MySQL based on Percona XtraDB Cluster does not support single node mode.

I think you need to use Percona Operator for MySQL based on Percona Server for MySQL.

I hope this helps.

Pep

Joshua_Sierles · July 12, 2024, 9:15am

Hmm, if it doesn’t support single nodes, why is that an option in Everest, which uses the XtraDB operator?

Also, what I’ve noticed in testing is that even with multi-node setups, this happens, so the first member of the StatefulSet never initializes. Surely this is a problem with my particular environment, but I haven’t figured out why yet.

Sergey_Pronin · July 12, 2024, 9:30am

@Joshua_Sierles I want to correct @Pep_Pla a bit. PXC supports one node. We even have cr-minimal.yaml as an example to deploy a single node.

And we have lots of use cases where users deploy PXC with a single node for their dev/test environments (as it is obviously not recommended to run a single node in production).

As for the issue you are facing - it needs deeper investigation. It does not happen for me in minikube or GKE (checked just now) + we have a test for this use case. Can you attach the operator logs?

Pep_Pla · July 13, 2024, 10:37am

It’s true; I typed too fast. Although you can run the PXC operator with one node, it is considered unsafe.

That’s why a CR option allows you to run in unsafe mode.

Check for allowUnsafeConfigurations in your cr file. It should be true. If it is false and the single cluster-pxc pod is killed, it does not restart properly.

I hope this helps.

Joshua_Sierles · July 13, 2024, 12:50pm

Thanks. The issue must be with my k8s environment. I’ll post logs later. Meanwhile, I was able to get the newer MySQL operator working with a single node.

drew_datajoint · November 22, 2024, 5:52pm

Hi @Sergey_Pronin ,
Hope you are doing well!
I’m experiencing the same issue, my environment:

kOps provisioned EC2 based generic k8s cluster
pxc-operator helm here
pxc-db helm CrVersion: 1.15.1

With all the default installation config, when I enable unsafeFlags.*, the operator pod still shows: Setting safe defaults, updating cluster size oldSize: 1 newSize: 3 here

After I set the pxc-operator’s image to perconalab/percona-xtradb-cluster-operator:main here, then the operator pod gets an error failing to launch PXC pods:

2024-11-22T17:23:34.821Z        ERROR   Reconciler error        {"controller": "pxc-controller", "namespace": "percona", "name": "percona-qa", "reconcileID": "6085e80c-4a9e-4d54-bb2f-87900f7266a9", "error": "wrong PXC options: check safe defaults: PXC size must be at least 3. Set spec.unsafeFlags.pxcSize to true to disable this check", "errorVerbose": "PXC size must be at least 3. Set spec.unsafeFlags.pxcSize to true to disable this check\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/apis/pxc/v1.(*PerconaXtraDBCluster).checkSafeDefaults\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/apis/pxc/v1/pxc_types.go:1202\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/apis/pxc/v1.(*PerconaXtraDBCluster).CheckNSetDefaults\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/apis/pxc/v1/pxc_types.go:880\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc.(*ReconcilePerconaXtraDBCluster).Reconcile\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc/controller.go:214\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.1/pkg/internal/controller/controller.go:116\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.1/pkg/internal/controller/controller.go:303\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.1/pkg/internal/controller/controller.go:263\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.1/pkg/internal/controller/controller.go:224\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1695\ncheck safe defaults\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/apis/pxc/v1.(*PerconaXtraDBCluster).CheckNSetDefaults\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/apis/pxc/v1/pxc_types.go:881\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc.(*ReconcilePerconaXtraDBCluster).Reconcile\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc/controller.go:214\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.1/pkg/internal/controller/controller.go:116\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.1/pkg/internal/controller/controller.go:303\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.1/pkg/internal/controller/controller.go:263\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.1/pkg/internal/controller/controller.go:224\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1695\nwrong PXC options\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc.(*ReconcilePerconaXtraDBCluster).Reconcile\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.1/pkg/internal/controller/controller.go:116\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.1/pkg/internal/controller/controller.go:303\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.1/pkg/internal/controller/controller.go:263\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.1/pkg/internal/controller/controller.go:224\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1695"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.1/pkg/internal/controller/controller.go:316
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.1/pkg/internal/controller/controller.go:263
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.1/pkg/internal/controller/controller.go:224

and I checked the PerconaXtraDBCluster object kubectl -n percona describe PerconaXtraDBCluster percona-qa, there is no spec.allowUnsafeConfigurations nor spec.unsafeFlags at all

Once I manually kubectl -n percona edit PerconaXtraDBCluster percona-qa and add spec.allowUnsafeConfigurations=true, then the PXC is able to launch and only launched with one node.

NAME                                       READY   STATUS    RESTARTS         AGE
percona-op-pxc-operator-67697548f5-2sfpj   1/1     Running   0                31m
percona-qa-haproxy-0                       2/2     Running   0                25m
percona-qa-haproxy-1                       2/2     Running   0                25m
percona-qa-haproxy-2                       2/2     Running   0                24m
percona-qa-pxc-0                           3/3     Running   0                25m

Slava_Sarzhan · November 27, 2024, 12:42pm

Hi @drew_datajoint, I have tested it today with PXC 1.15.1:

> helm install my-op percona/pxc-operator --namespace pxc
❯ helm install my-db percona/pxc-db  --namespace pxc --set "unsafeFlags.pxcSize=true" --set "pxc.size=1"

❯ kubectl get pods -n pxc
NAME                                  READY   STATUS    RESTARTS   AGE
my-db-pxc-db-haproxy-0                2/2     Running   0          3m14s
my-db-pxc-db-haproxy-1                2/2     Running   0          114s
my-db-pxc-db-haproxy-2                2/2     Running   0          83s
my-db-pxc-db-pxc-0                    3/3     Running   0          3m14s
my-op-pxc-operator-6868dd7fdc-qlwcj   1/1     Running   0          6m20s

As you can see, only one PXC pod was created. Could you please recheck it from your end.

Topic		Replies	Views
Is it possible to deploy pxc on one single node? Percona Operator for MySQL	1	871	August 20, 2021
Percona XtraDB, Question to the experts!!! ;) Percona XtraDB Cluster 5.x	3	832	August 10, 2015
Can't set up 1 node pxc on kubernetes cluster Percona XtraDB Cluster 5.x mysql , percona	2	462	February 1, 2024
Xtradb operator - mysql multimaster and replica Percona XtraDB Cluster 8.x percona	9	23	April 11, 2025
Why is the Operator for PXC (Glaera) only? Percona Operator for MySQL	2	518	April 7, 2021

Percona XtraDB operator with single node detects itself as peer

Related topics