Description:
Hello,
I have deployed two percona-server instances with:
unsafeFlags:
mysqlSize: true
proxy: true
proxySize: true
orchestrator: true
orchestratorSize: true
mysql:
size: 2
imagePullSecrets:
- name: regcred
clusterType: async
volumeSpec:
pvc:
storageClass: "local-path"
accessMode: ReadWriteOnce
resources:
requests:
storage: 20Gi
resources:
requests:
memory: "12Gi"
cpu: "6"
limits:
memory: "12Gi"
cpu: "6"
Everything is started successfully. Now I am trying to test failover. I am killing the first percona-server pod (that was primary) and see that the replica was promoted as master in the orchestrator UI. I expected (sorry if I am missing something) that after restart of the previous primary it will be promoted as replica but instead it is in the CrashLoopBackOff loop and the Perconaservermysqls has reconciliation error - replication: failed to discover cluster: driver: bad connection.
Looking at the percona-server logs (attached below) I have a feeling that it is restarted as primary again and it is confirmed by orchestrator, at least it is labeled as master that can not recover:
This expectation that primary and replica should change their roles and replica become secondary (and writable) and the previous primary after restart should start to be replica - is it valid expectation in general? Maybe manual kill of pod the ugly variant to test it and for example normal upgrade of cluster would work, but what if OOMKiller for example.
Do you have any hints what I am missing?
Steps to Reproduce:
Setup 2 instance with async replication and kill the first pod.
Version:
- Percona Server - percona/percona-server:8.0.36-28
- Percona MySQL operator percona/percona-server-mysql-operator:0.8.0
- Orchestrator - percona/percona-orchestrator:3.2.6-12
Logs:
Logs of the percona-operator-mysql after primary restart:
2025-01-27T12:48:23.225Z INFO Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes reqeueuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler {"controller": "ps-controller", "controllerGroup": "ps.percona.com", "controllerKind": "PerconaServerMySQL", "PerconaServerMySQL": {"name":"mysql-cluster-ps-db","namespace":"testing"}, "namespace": "testing", "name": "mysql-cluster-ps-db", "reconcileID": "ac664d54-8132-461f-a8db-163b867120dc"}
2025-01-27T12:48:23.225Z ERROR Reconciler error {"controller": "ps-controller", "controllerGroup": "ps.percona.com", "controllerKind": "PerconaServerMySQL", "PerconaServerMySQL": {"name":"mysql-cluster-ps-db","namespace":"testing"}, "namespace": "testing", "name": "mysql-cluster-ps-db", "reconcileID": "ac664d54-8132-461f-a8db-163b867120dc", "error": "reconcile: replication: failed to discover cluster: driver: bad connection", "errorVerbose": "driver: bad connection\ngithub.com/percona/percona-server-mysql-operator/pkg/orchestrator.Discover\n\t/go/src/github.com/percona/percona-server-mysql-operator/pkg/orchestrator/client.go:228\ngithub.com/percona/percona-server-mysql-operator/pkg/controller/ps.(*PerconaServerMySQLReconciler).reconcileReplication\n\t/go/src/github.com/percona/percona-server-mysql-operator/pkg/controller/ps/controller.go:806\ngithub.com/percona/percona-server-mysql-operator/pkg/controller/ps.(*PerconaServerMySQLReconciler).doReconcile\n\t/go/src/github.com/percona/percona-server-mysql-operator/pkg/controller/ps/controller.go:383\ngithub.com/percona/percona-server-mysql-operator/pkg/controller/ps.(*PerconaServerMySQLReconciler).Reconcile\n\t/go/src/github.com/percona/percona-server-mysql-operator/pkg/controller/ps/controller.go:125\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:222\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1695\nfailed to discover cluster\ngithub.com/percona/percona-server-mysql-operator/pkg/controller/ps.(*PerconaServerMySQLReconciler).reconcileReplication\n\t/go/src/github.com/percona/percona-server-mysql-operator/pkg/controller/ps/controller.go:815\ngithub.com/percona/percona-server-mysql-operator/pkg/controller/ps.(*PerconaServerMySQLReconciler).doReconcile\n\t/go/src/github.com/percona/percona-server-mysql-operator/pkg/controller/ps/controller.go:383\ngithub.com/percona/percona-server-mysql-operator/pkg/controller/ps.(*PerconaServerMySQLReconciler).Reconcile\n\t/go/src/github.com/percona/percona-server-mysql-operator/pkg/controller/ps/controller.go:125\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:222\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1695\nreplication\ngithub.com/percona/percona-server-mysql-operator/pkg/controller/ps.(*PerconaServerMySQLReconciler).doReconcile\n\t/go/src/github.com/percona/percona-server-mysql-operator/pkg/controller/ps/controller.go:384\ngithub.com/percona/percona-server-mysql-operator/pkg/controller/ps.(*PerconaServerMySQLReconciler).Reconcile\n\t/go/src/github.com/percona/percona-server-mysql-operator/pkg/controller/ps/controller.go:125\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:222\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1695\nreconcile\ngithub.com/percona/percona-server-mysql-operator/pkg/controller/ps.(*PerconaServerMySQLReconciler).Reconcile\n\t/go/src/github.com/percona/percona-server-mysql-operator/pkg/controller/ps/controller.go:126\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:222\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1695"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:324
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:261
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:222
Logs of the percona-server instance which is in the CrashLoopBackOff loop:
+ '[' m = - ']'
+ wantHelp=
+ for arg in "$@"
+ case "$arg" in
+ CFG=/etc/my.cnf.d/node.cnf
+ TLS_DIR=/etc/mysql/mysql-tls-secret
+ CUSTOM_CONFIG_FILES=("/etc/mysql/config/auto-config.cnf" "/etc/mysql/config/my-config.cnf" "/etc/mysql/config/my-secret.cnf")
++ mysqld -V
++ awk -F. '{print $1"."$2}'
++ awk '{print $3}'
+ MYSQL_VERSION=8.0
+ '[' 8.0 '!=' 8.0 ']'
+ '[' mysqld = mysqld -a -z '' ']'
+ _check_config mysqld
+ toRun=("$@" --verbose --help)
++ mysqld --verbose --help
+ errors=
++ _get_config datadir mysqld
++ local conf=datadir
++ shift
++ awk '$1 == "datadir" && /^[^ \t]/ { sub(/^[^ \t]+[ \t]+/, ""); print; exit }'
+++ mktemp -u
++ mysqld --verbose --help --log-bin-index=/tmp/tmp.06itgvWPNB
+ DATADIR=/var/lib/mysql/
++ _get_tmpdir /var/lib/mysql//mysql-tmpdir
++ local defaul_value=/var/lib/mysql//mysql-tmpdir
++ local tmpdir_path=
+++ _get_cnf_config mysqld tmpdir ''
+++ local group=mysqld
+++ local var=tmpdir
+++ local reval=
++++ my_print_defaults mysqld
++++ awk -F= '{st=index($0,"="); cur=$0; if ($1 ~ /_/) { gsub(/_/,"-",$1);} if (st != 0) { print $1"="substr(cur,st+1) } else { print cur }}'
++++ grep -- --tmpdir=
++++ cut -d= -f2-
++++ tail -1
+++ reval=
+++ [[ -z '' ]]
+++ reval=
+++ echo ''
++ tmpdir_path=
++ [[ -z '' ]]
+++ _get_cnf_config xtrabackup tmpdir ''
+++ local group=xtrabackup
+++ local var=tmpdir
+++ local reval=
++++ my_print_defaults xtrabackup
++++ awk -F= '{st=index($0,"="); cur=$0; if ($1 ~ /_/) { gsub(/_/,"-",$1);} if (st != 0) { print $1"="substr(cur,st+1) } else { print cur }}'
++++ grep -- --tmpdir=
++++ cut -d= -f2-
++++ tail -1
+++ reval=
+++ [[ -z '' ]]
+++ reval=
+++ echo ''
++ tmpdir_path=
++ [[ -z '' ]]
++ tmpdir_path=/var/lib/mysql//mysql-tmpdir
++ echo /var/lib/mysql//mysql-tmpdir
+ TMPDIR=/var/lib/mysql//mysql-tmpdir
+ rm -rfv /var/lib/mysql//mysql-tmpdir
+ create_default_cnf
++ hostname -I
++ awk '{print $1}'
+ POD_IP=10.42.163.186
+ [[ mysql-cluster-ps-db-mysql-0 =~ -xb- ]]
++ hostname -f
++ cut -d. -f2
+ CLUSTER_NAME=mysql-cluster-ps-db-mysql
+ SERVER_NUM=0
+ SERVER_ID=69457480
+ FQDN=mysql-cluster-ps-db-mysql-0.mysql-cluster-ps-db-mysql.testing
+ echo '[mysqld]'
+ sed -i '/\[mysqld\]/a read_only=ON' /etc/my.cnf.d/node.cnf
+ sed -i '/\[mysqld\]/a server_id=69457480' /etc/my.cnf.d/node.cnf
+ sed -i '/\[mysqld\]/a admin-address=10.42.163.186' /etc/my.cnf.d/node.cnf
+ sed -i '/\[mysqld\]/a report_host=mysql-cluster-ps-db-mysql-0.mysql-cluster-ps-db-mysql.testing' /etc/my.cnf.d/node.cnf
+ sed -i '/\[mysqld\]/a report_port=3306' /etc/my.cnf.d/node.cnf
+ sed -i '/\[mysqld\]/a gtid-mode=ON' /etc/my.cnf.d/node.cnf
+ sed -i '/\[mysqld\]/a enforce-gtid-consistency=ON' /etc/my.cnf.d/node.cnf
+ sed -i '/\[mysqld\]/a log_error_verbosity=3' /etc/my.cnf.d/node.cnf
+ sed -i '/\[mysqld\]/a plugin-load-add=clone=mysql_clone.so' /etc/my.cnf.d/node.cnf
+ [[ -d /etc/mysql/mysql-tls-secret ]]
+ sed -i '/\[mysqld\]/a ssl_ca=/etc/mysql/mysql-tls-secret/ca.crt' /etc/my.cnf.d/node.cnf
+ sed -i '/\[mysqld\]/a ssl_cert=/etc/mysql/mysql-tls-secret/tls.crt' /etc/my.cnf.d/node.cnf
+ sed -i '/\[mysqld\]/a ssl_key=/etc/mysql/mysql-tls-secret/tls.key' /etc/my.cnf.d/node.cnf
/etc/mysql/config/auto-config.cnf
+ for f in "${CUSTOM_CONFIG_FILES[@]}"
+ echo /etc/mysql/config/auto-config.cnf
+ '[' -f /etc/mysql/config/auto-config.cnf ']'
+ cat /etc/mysql/config/auto-config.cnf
+ echo
+ for f in "${CUSTOM_CONFIG_FILES[@]}"
+ echo /etc/mysql/config/my-config.cnf
+ '[' -f /etc/mysql/config/my-config.cnf ']'
/etc/mysql/config/my-config.cnf
+ cat /etc/mysql/config/my-config.cnf
+ echo
+ for f in "${CUSTOM_CONFIG_FILES[@]}"
+ echo /etc/mysql/config/my-secret.cnf
+ '[' -f /etc/mysql/config/my-secret.cnf ']'
/etc/mysql/config/my-secret.cnf
+ '[' '!' -d /var/lib/mysql//mysql ']'
+ load_group_replication_plugin
++ hostname -I
++ awk '{print $1}'
+ POD_IP=10.42.163.186
+ sed -i '/\[mysqld\]/a plugin_load_add=group_replication.so' /etc/my.cnf.d/node.cnf
+ sed -i '/\[mysqld\]/a group_replication_exit_state_action=ABORT_SERVER' /etc/my.cnf.d/node.cnf
+ ensure_read_only
+ sed -i '/\[mysqld\]/a read_only=ON' /etc/my.cnf.d/node.cnf
+ sed -i '/\[mysqld\]/a super_read_only=ON' /etc/my.cnf.d/node.cnf
+ '[' -n '' ']'
+ [[ -f /var/lib/mysql/full-cluster-crash ]]
+ recovery_file=/var/lib/mysql/sleep-forever
+ '[' -f /var/lib/mysql/sleep-forever ']'
+ exec mysqld
2025-01-27T12:54:09.898555Z 0 [Note] [MY-013667] [Server] Error-log destination "stderr" is not a file. Can not restore error log messages from previous run.
2025-01-27T12:54:09.894361Z 0 [Warning] [MY-011068] [Server] The syntax '--skip-host-cache' is deprecated and will be removed in a future release. Please use SET GLOBAL host_cache_size=0 instead.
2025-01-27T12:54:09.896653Z 0 [Warning] [MY-010097] [Server] Insecure configuration for --secure-log-path: Current value does not restrict location of generated files. Consider setting it to a valid, non-empty path.
2025-01-27T12:54:09.896712Z 0 [Warning] [MY-010918] [Server] 'default_authentication_plugin' is deprecated and will be removed in a future release. Please use authentication_policy instead.
2025-01-27T12:54:09.896722Z 0 [Note] [MY-013932] [Server] BuildID[sha1]=d18f29188cae66133efa80e6eb798f0bc921812c
2025-01-27T12:54:09.896732Z 0 [Note] [MY-010949] [Server] Basedir set to /usr/.
2025-01-27T12:54:09.896744Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.36-28) starting as process 1
2025-01-27T12:54:09.897719Z 0 [Warning] [MY-013242] [Server] --character-set-server: 'utf8' is currently an alias for the character set UTF8MB3, but will be an alias for UTF8MB4 in a future release. Please consider using UTF8MB4 in order to be unambiguous.
2025-01-27T12:54:09.897732Z 0 [Warning] [MY-013244] [Server] --collation-server: 'utf8mb3_general_ci' is a collation of the deprecated character set UTF8MB3. Please consider using UTF8MB4 with an appropriate collation instead.
2025-01-27T12:54:09.901419Z 0 [Note] [MY-012366] [InnoDB] Using Linux native AIO
2025-01-27T12:54:09.901466Z 0 [Warning] [MY-013907] [InnoDB] Deprecated configuration parameters innodb_log_file_size and/or innodb_log_files_in_group have been used to compute innodb_redo_log_capacity=4294967296. Please use innodb_redo_log_capacity instead.
2025-01-27T12:54:09.901656Z 0 [Note] [MY-010747] [Server] Plugin 'FEDERATED' is disabled.
2025-01-27T12:54:09.901756Z 0 [Note] [MY-010747] [Server] Plugin 'ndbcluster' is disabled.
2025-01-27T12:54:09.901782Z 0 [Note] [MY-010747] [Server] Plugin 'ndbinfo' is disabled.
2025-01-27T12:54:09.901794Z 0 [Note] [MY-010747] [Server] Plugin 'ndb_transid_mysql_connection_map' is disabled.
2025-01-27T12:54:09.903535Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2025-01-27T12:54:09.903659Z 1 [Note] [MY-013547] [InnoDB] Atomic write disabled
2025-01-27T12:54:09.903765Z 1 [Note] [MY-012932] [InnoDB] PUNCH HOLE support available
2025-01-27T12:54:09.903818Z 1 [Note] [MY-012944] [InnoDB] Uses event mutexes
2025-01-27T12:54:09.903840Z 1 [Note] [MY-012945] [InnoDB] GCC builtin __atomic_thread_fence() is used for memory barrier
2025-01-27T12:54:09.903878Z 1 [Note] [MY-012948] [InnoDB] Compressed tables use zlib 1.2.13
2025-01-27T12:54:09.916447Z 1 [Note] [MY-012951] [InnoDB] Using hardware accelerated crc32 and polynomial multiplication.
2025-01-27T12:54:09.917793Z 1 [Note] [MY-012203] [InnoDB] Directories to scan './'
2025-01-27T12:54:09.917897Z 1 [Note] [MY-012204] [InnoDB] Scanning './'
2025-01-27T12:54:09.926213Z 1 [Note] [MY-012208] [InnoDB] Completed space ID check of 6 files.
2025-01-27T12:54:09.929027Z 1 [Note] [MY-012955] [InnoDB] Initializing buffer pool, total size = 8.000000G, instances = 8, chunk size =128.000000M
2025-01-27T12:54:12.159606Z 1 [Note] [MY-012957] [InnoDB] Completed initialization of buffer pool
2025-01-27T12:54:12.221124Z 0 [Note] [MY-011952] [InnoDB] If the mysqld execution user is authorized, page cleaner and LRU manager thread priority can be changed. See the man page of setpriority().
2025-01-27T12:54:12.654545Z 1 [Note] [MY-013883] [InnoDB] The latest found checkpoint is at lsn = 42754318 in redo log file ./#innodb_redo/#ib_redo0.
2025-01-27T12:54:12.654711Z 1 [Note] [MY-013086] [InnoDB] Starting to parse redo log at lsn = 42754090, whereas checkpoint_lsn = 42754318 and start_lsn = 42754048
2025-01-27T12:54:12.694680Z 1 [Note] [MY-013083] [InnoDB] Log background threads are being started...
2025-01-27T12:54:12.696173Z 1 [Note] [MY-012532] [InnoDB] Applying a batch of 0 redo log records ...
2025-01-27T12:54:12.696203Z 1 [Note] [MY-012535] [InnoDB] Apply batch completed!
2025-01-27T12:54:12.697863Z 1 [Note] [MY-013252] [InnoDB] Using undo tablespace './undo_001'.
2025-01-27T12:54:12.699695Z 1 [Note] [MY-013252] [InnoDB] Using undo tablespace './undo_002'.
2025-01-27T12:54:12.701403Z 1 [Note] [MY-012910] [InnoDB] Opened 2 existing undo tablespaces.
2025-01-27T12:54:12.701473Z 1 [Note] [MY-011980] [InnoDB] GTID recovery trx_no: 46864
2025-01-27T12:54:12.872504Z 1 [Note] [MY-013776] [InnoDB] Parallel initialization of rseg complete
2025-01-27T12:54:12.872607Z 1 [Note] [MY-013777] [InnoDB] Time taken to initialize rseg using 4 thread: 171129 ms.
2025-01-27T12:54:12.872857Z 1 [Note] [MY-012923] [InnoDB] Creating shared tablespace for temporary tables
2025-01-27T12:54:12.873202Z 1 [Note] [MY-012265] [InnoDB] Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2025-01-27T12:54:12.933443Z 1 [Note] [MY-012266] [InnoDB] File './ibtmp1' size is now 12 MB.
2025-01-27T12:54:12.933991Z 1 [Note] [MY-013627] [InnoDB] Scanning temp tablespace dir:'./#innodb_temp/'
2025-01-27T12:54:12.983083Z 1 [Note] [MY-013018] [InnoDB] Created 128 and tracked 128 new rollback segment(s) in the temporary tablespace. 128 are now active.
2025-01-27T12:54:12.984486Z 1 [Note] [MY-012976] [InnoDB] Percona XtraDB (http://www.percona.com) 8.0.36-28 started; log sequence number 42754328
2025-01-27T12:54:12.986945Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2025-01-27T12:54:13.017662Z 1 [Note] [MY-011089] [Server] Data dictionary restarting version '80023'.
2025-01-27T12:54:13.225403Z 1 [Note] [MY-012357] [InnoDB] Reading DD tablespace files
2025-01-27T12:54:13.226659Z 1 [Note] [MY-012356] [InnoDB] Scanned 8 tablespaces. Validated 8.
2025-01-27T12:54:13.261063Z 1 [Note] [MY-010006] [Server] Using data dictionary with version '80023'.
2025-01-27T12:54:13.272495Z 0 [ERROR] [MY-013180] [Server] Function 'group_replication' already exists.
2025-01-27T12:54:13.272639Z 0 [ERROR] [MY-010736] [Server] Couldn't load plugin named 'group_replication' with soname 'group_replication.so'.
2025-01-27T12:54:13.284610Z 0 [Note] [MY-011332] [Server] Plugin mysqlx reported: 'IPv6 is available'
2025-01-27T12:54:13.288212Z 0 [Note] [MY-011323] [Server] Plugin mysqlx reported: 'X Plugin ready for connections. bind-address: '::' port: 33060'
2025-01-27T12:54:13.288410Z 0 [Note] [MY-011323] [Server] Plugin mysqlx reported: 'X Plugin ready for connections. socket: '/var/lib/mysql/mysqlx.sock''
2025-01-27T12:54:13.308212Z 0 [ERROR] [MY-011685] [Repl] Plugin group_replication reported: 'The group_replication_group_name option is mandatory'
2025-01-27T12:54:13.308441Z 0 [ERROR] [MY-011660] [Repl] Plugin group_replication reported: 'Unable to start Group Replication on boot'
2025-01-27T12:54:13.371572Z 0 [Note] [MY-010902] [Server] Thread priority attribute setting in Resource Group SQL shall be ignored due to unsupported platform or insufficient privilege.
2025-01-27T12:54:13.379859Z 0 [Note] [MY-010856] [Server] Failed to open the crashed binlog file when source server is recovering it.
2025-01-27T12:54:13.407312Z 0 [Note] [MY-013911] [Server] Crash recovery finished in binlog engine. No attempts to commit, rollback or prepare any transactions.
2025-01-27T12:54:13.407348Z 0 [Note] [MY-013911] [Server] Crash recovery finished in InnoDB engine. No attempts to commit, rollback or prepare any transactions.
2025-01-27T12:54:13.411367Z 0 [Note] [MY-012487] [InnoDB] DDL log recovery : begin
2025-01-27T12:54:13.412297Z 0 [Note] [MY-012488] [InnoDB] DDL log recovery : end
2025-01-27T12:54:13.412682Z 0 [Note] [MY-011946] [InnoDB] Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
2025-01-27T12:54:13.428940Z 0 [Note] [MY-012922] [InnoDB] Waiting for purge to start
2025-01-27T12:54:13.535614Z 0 [Note] [MY-010303] [Server] Skipping generation of SSL certificates as options related to SSL are specified.
2025-01-27T12:54:13.542110Z 0 [Warning] [MY-010068] [Server] CA certificate /etc/mysql/mysql-tls-secret/ca.crt is self signed.
2025-01-27T12:54:13.542191Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel.
2025-01-27T12:54:13.589754Z 0 [Warning] [MY-013595] [Server] Failed to initialize TLS for channel: mysql_admin. See below for the description of exact issue.
2025-01-27T12:54:13.589822Z 0 [Warning] [MY-010069] [Server] Failed to set up SSL because of the following SSL library error: SSL context is not usable without certificate and private key
2025-01-27T12:54:13.589835Z 0 [System] [MY-013603] [Server] No TLS configuration was given for channel mysql_admin; re-using TLS configuration of channel mysql_main.
2025-01-27T12:54:13.590366Z 0 [Note] [MY-010308] [Server] Skipping generation of RSA key pair through --sha256_password_auto_generate_rsa_keys as key files are present in data directory.
2025-01-27T12:54:13.590785Z 0 [Note] [MY-010308] [Server] Skipping generation of RSA key pair through --caching_sha2_password_auto_generate_rsa_keys as key files are present in data directory.
2025-01-27T12:54:13.593723Z 0 [Note] [MY-010252] [Server] Server hostname (bind-address): '10.42.163.186'; port: 33062
2025-01-27T12:54:13.593756Z 0 [Note] [MY-010264] [Server] - '10.42.163.186' resolves to '10.42.163.186';
2025-01-27T12:54:13.593785Z 0 [Note] [MY-010251] [Server] Server socket created on IP: '10.42.163.186'.
2025-01-27T12:54:13.593810Z 0 [Note] [MY-010252] [Server] Server hostname (bind-address): '0.0.0.0'; port: 3306
2025-01-27T12:54:13.593816Z 0 [Note] [MY-010264] [Server] - '0.0.0.0' resolves to '0.0.0.0';
2025-01-27T12:54:13.593822Z 0 [Note] [MY-010251] [Server] Server socket created on IP: '0.0.0.0'.
2025-01-27T12:54:13.640934Z 0 [Note] [MY-011025] [Repl] Failed to start replica threads for channel ''.
2025-01-27T12:54:13.641666Z 0 [Note] [MY-011946] [InnoDB] Buffer pool(s) load completed at 250127 12:54:13
2025-01-27T12:54:13.643143Z 0 [Note] [MY-011240] [Server] Plugin mysqlx reported: 'Using SSL configuration from MySQL Server'
2025-01-27T12:54:13.643244Z 7 [Note] [MY-010051] [Server] Event Scheduler: scheduler thread started with id 7
2025-01-27T12:54:13.645896Z 0 [Note] [MY-011243] [Server] Plugin mysqlx reported: 'Using OpenSSL for TLS connections'
2025-01-27T12:54:13.646139Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: '::' port: 33060, socket: /var/lib/mysql/mysqlx.sock
2025-01-27T12:54:13.646182Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.36-28' socket: '/var/lib/mysql/mysql.sock' port: 3306 Percona Server (GPL), Release 28, Revision 47601f19.
2025-01-27T12:54:13.646267Z 0 [System] [MY-013292] [Server] Admin interface ready for connections, address: '10.42.163.186' port: 33062
2025-01-27T12:54:33.436774Z 10 [Warning] [MY-013360] [Server] Plugin mysql_native_password reported: ''mysql_native_password' is deprecated and will be removed in a future release. Please use caching_sha2_password instead'
2025-01-27T12:54:33.595223Z 11 [Warning] [MY-013360] [Server] Plugin mysql_native_password reported: ''mysql_native_password' is deprecated and will be removed in a future release. Please use caching_sha2_password instead'
2025-01-27T12:54:33.758292Z 12 [Warning] [MY-013360] [Server] Plugin mysql_native_password reported: ''mysql_native_password' is deprecated and will be removed in a future release. Please use caching_sha2_password instead'
2025-01-27T12:54:33.935306Z 0 [System] [MY-013172] [Server] Received SHUTDOWN from user <via user signal>. Shutting down mysqld (Version: 8.0.36-28).
2025-01-27T12:54:33.937162Z 0 [Note] [MY-010067] [Server] Giving 4 client threads a chance to die gracefully
2025-01-27T12:54:33.937229Z 0 [Note] [MY-010117] [Server] Shutting down replica threads
2025-01-27T12:54:33.937265Z 0 [Note] [MY-010054] [Server] Event Scheduler: Killing the scheduler thread, thread id 7
2025-01-27T12:54:33.937302Z 0 [Note] [MY-010050] [Server] Event Scheduler: Waiting for the scheduler thread to reply
2025-01-27T12:54:33.937567Z 0 [Note] [MY-010048] [Server] Event Scheduler: Stopped
2025-01-27T12:54:35.937825Z 0 [Note] [MY-010118] [Server] Forcefully disconnecting 2 remaining clients
2025-01-27T12:54:35.938786Z 0 [Note] [MY-010043] [Server] Event Scheduler: Purging the queue. 0 events
2025-01-27T12:54:35.940416Z 0 [Note] [MY-012330] [InnoDB] FTS optimize thread exiting.
2025-01-27T12:54:36.241198Z 0 [Note] [MY-010120] [Server] Binlog end
2025-01-27T12:54:36.248599Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'clone'
2025-01-27T12:54:36.248685Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'group_replication'
2025-01-27T12:54:36.248769Z 0 [Note] [MY-011665] [Repl] Plugin group_replication reported: 'All Group Replication server observers have been successfully unregistered'
2025-01-27T12:54:36.248851Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'mysqlx'
2025-01-27T12:54:36.249897Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'mysqlx_cache_cleaner'
2025-01-27T12:54:36.249946Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'ngram'
2025-01-27T12:54:36.249965Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'BLACKHOLE'
2025-01-27T12:54:36.249987Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'ARCHIVE'
2025-01-27T12:54:36.250004Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'TempTable'
2025-01-27T12:54:36.250020Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'MRG_MYISAM'
2025-01-27T12:54:36.250048Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'MyISAM'
2025-01-27T12:54:36.250082Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'INNODB_SESSION_TEMP_TABLESPACES'
2025-01-27T12:54:36.250101Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'INNODB_CACHED_INDEXES'
2025-01-27T12:54:36.250118Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'INNODB_VIRTUAL'
2025-01-27T12:54:36.250135Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'INNODB_COLUMNS'
2025-01-27T12:54:36.250151Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'INNODB_TABLESPACES'
2025-01-27T12:54:36.250190Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'INNODB_INDEXES'
2025-01-27T12:54:36.250206Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'INNODB_TABLESTATS'
2025-01-27T12:54:36.250223Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'INNODB_TABLES'
2025-01-27T12:54:36.250240Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'INNODB_FT_INDEX_TABLE'
2025-01-27T12:54:36.250256Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'INNODB_FT_INDEX_CACHE'
2025-01-27T12:54:36.250274Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'INNODB_FT_CONFIG'
2025-01-27T12:54:36.250290Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'INNODB_FT_BEING_DELETED'
2025-01-27T12:54:36.250306Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'INNODB_FT_DELETED'
2025-01-27T12:54:36.250323Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'INNODB_FT_DEFAULT_STOPWORD'
2025-01-27T12:54:36.250340Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'INNODB_METRICS'
2025-01-27T12:54:36.250356Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'INNODB_TEMP_TABLE_INFO'
2025-01-27T12:54:36.250373Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'INNODB_BUFFER_POOL_STATS'
2025-01-27T12:54:36.250389Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'INNODB_BUFFER_PAGE_LRU'
2025-01-27T12:54:36.250406Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'INNODB_BUFFER_PAGE'
2025-01-27T12:54:36.250421Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'INNODB_CMP_PER_INDEX_RESET'
2025-01-27T12:54:36.250437Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'INNODB_CMP_PER_INDEX'
2025-01-27T12:54:36.250453Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'INNODB_CMPMEM_RESET'
2025-01-27T12:54:36.250469Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'INNODB_CMPMEM'
2025-01-27T12:54:36.250486Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'INNODB_CMP_RESET'
2025-01-27T12:54:36.250502Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'INNODB_CMP'
2025-01-27T12:54:36.250529Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'INNODB_TRX'
2025-01-27T12:54:36.250546Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'InnoDB'
2025-01-27T12:54:36.250608Z 0 [Note] [MY-013072] [InnoDB] Starting shutdown...
2025-01-27T12:54:36.250869Z 0 [Note] [MY-011944] [InnoDB] Dumping buffer pool(s) to /var/lib/mysql/ib_buffer_pool
2025-01-27T12:54:36.251474Z 0 [Note] [MY-011944] [InnoDB] Buffer pool(s) dump completed at 250127 12:54:36
2025-01-27T12:54:36.264879Z 0 [Note] [MY-013084] [InnoDB] Log background threads are being closed...
2025-01-27T12:54:36.960710Z 0 [Note] [MY-012980] [InnoDB] Shutdown completed; log sequence number 42841886
2025-01-27T12:54:36.960840Z 0 [Note] [MY-012255] [InnoDB] Removed temporary tablespace data file: "ibtmp1"
2025-01-27T12:54:36.960872Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'MEMORY'
2025-01-27T12:54:36.960885Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'CSV'
2025-01-27T12:54:36.960894Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'PERFORMANCE_SCHEMA'
2025-01-27T12:54:36.960923Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'daemon_keyring_proxy_plugin'
2025-01-27T12:54:36.960945Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'sha2_cache_cleaner'
2025-01-27T12:54:36.960954Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'caching_sha2_password'
2025-01-27T12:54:36.960963Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'sha256_password'
2025-01-27T12:54:36.960969Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'mysql_native_password'
2025-01-27T12:54:36.962072Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'binlog'
2025-01-27T12:54:36.963940Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.36-28) Percona Server (GPL), Release 28, Revision 47601f19.
In the orchestrator logs I see:
2025-01-27 12:58:34 WARNING DiscoverInstance(mysql-cluster-ps-db-mysql-0.mysql-cluster-ps-db-mysql.testing:3306) instance is nil in 0.013s (Backend: 0.004s, Instance: 0.009s), error=dial tcp: lookup mysql-cluster-ps-db-mysql-0.mysql-cluster-ps-db-mysql.testing on 10.43.0.10:53: no such host