Documentation followed: Percona Operator for MySQL based on Percona Server for MySQL
Github Repo: percona-server-mysql-operator/deploy at v0.6.0 · percona/percona-server-mysql-operator · GitHub
Setup Environment: Managed Kubernetes where container cannot runAsRoot and cannot run with non numeric user (mysql). podSecurityContext and containerSecurityContext has been used to run containers.
With v0.6.0, i have successfully deployed the operator using kubectl apply -d build/bundle.yaml
But while deploying mysql cluster with group-replication the first pod gets shutdown with error group_replication_group_name not set.
2024-02-01T11:47:25.083094Z 0 [ERROR] [MY-011685] [Repl] Plugin group_replication reported: 'The group_replication_group_name option is mandatory'
2024-02-01T11:47:25.083122Z 0 [ERROR] [MY-011660] [Repl] Plugin group_replication reported: 'Unable to start Group Replication on boot'
From the above github repo i am unable to figure out how to add any mysql group replication variables to my MySQL cluster yamls. Below is my cr.yaml to deploy PerconaServerMySQL
apiVersion: ps.percona.com/v1alpha1
kind: PerconaServerMySQL
metadata:
name: cluster1
finalizers:
- delete-mysql-pods-in-order
spec:
allowUnsafeConfigurations: false
crVersion: 0.6.0
secretsName: cluster1-secrets
sslSecretName: cluster1-ssl
updateStrategy: SmartUpdate
mysql:
clusterType: group-replication
autoRecovery: true
image: percona/percona-server:8.0.33-25
imagePullPolicy: Always
podSecurityContext:
fsGroup: 2000
containerSecurityContext:
runAsUser: 999
size: 3
resources:
requests:
memory: 1G
limits:
memory: 2G
affinity:
antiAffinityTopologyKey: "kubernetes.io/hostname"
volumeSpec:
persistentVolumeClaim:
storageClassName: ebs-premium-1a-ext4
resources:
requests:
storage: 2G
proxy:
haproxy:
enabled: true
size: 3
image: percona/haproxy:2.8.1
imagePullPolicy: Always
resources:
requests:
memory: 1G
cpu: 600m
affinity:
antiAffinityTopologyKey: "kubernetes.io/hostname"
router:
enabled: false
image: percona/percona-mysql-router:8.0.33
imagePullPolicy: Always
size: 3
resources:
requests:
memory: 256M
limits:
memory: 256M
affinity:
antiAffinityTopologyKey: "kubernetes.io/hostname"
orchestrator:
enabled: false
image: percona/percona-orchestrator:3.2.6-9
imagePullPolicy: Always
size: 3
affinity:
antiAffinityTopologyKey: "kubernetes.io/hostname"
resources:
requests:
memory: 128M
limits:
memory: 256M
volumeSpec:
persistentVolumeClaim:
storageClassName: ebs-premium-1a-ext4
resources:
requests:
storage: 1G
pmm:
enabled: false
image: percona/pmm-client:2.39.0
imagePullPolicy: Always
resources:
requests:
memory: 150M
cpu: 300m
serverHost: monitoring-service
serverUser: admin
backup:
enabled: false
image: percona/percona-xtrabackup:8.0.33-27
imagePullPolicy: Always
storages:
s3-us-west:
type: s3
verifyTLS: true
s3:
bucket: S3-BACKUP-BUCKET-NAME-HERE
credentialsSecret: cluster1-s3-credentials
region: us-west-2
toolkit:
image: percona/percona-server-mysql-operator:0.6.0-toolkit
imagePullPolicy: Always
Adding detailed logs:
+ rm /var/lib/mysql/bootstrap.lock
+ echo
+ echo 'MySQL init process done. Ready for start up.'
+ echo
+ load_group_replication_plugin
MySQL init process done. Ready for start up.
++ hostname -I
++ awk '{print $1}'
+ POD_IP=10.49.203.6
+ sed -i '/\[mysqld\]/a plugin_load_add=group_replication.so' /etc/my.cnf.d/node.cnf
+ sed -i '/\[mysqld\]/a group_replication_exit_state_action=ABORT_SERVER' /etc/my.cnf.d/node.cnf
+ ensure_read_only
+ sed -i '/\[mysqld\]/a read_only=ON' /etc/my.cnf.d/node.cnf
+ sed -i '/\[mysqld\]/a super_read_only=ON' /etc/my.cnf.d/node.cnf
+ '[' -n '' ']'
+ [[ -f /var/lib/mysql/full-cluster-crash ]]
+ recovery_file=/var/lib/mysql/sleep-forever
+ '[' -f /var/lib/mysql/sleep-forever ']'
+ exec mysqld
2024-02-01T11:47:08.376105Z 0 [Note] [MY-013667] [Server] Error-log destination "stderr" is not a file. Can not restore error log messages from previous run.
2024-02-01T11:47:08.282517Z 0 [Warning] [MY-011068] [Server] The syntax '--skip-host-cache' is deprecated and will be removed in a future release. Please use SET GLOBAL host_cache_size=0 instead.
2024-02-01T11:47:08.284757Z 0 [Warning] [MY-010097] [Server] Insecure configuration for --secure-log-path: Current value does not restrict location of generated files. Consider setting it to a valid, non-empty path.
2024-02-01T11:47:08.284810Z 0 [Note] [MY-013932] [Server] BuildID[sha1]=cde83959eaef64440e90d44d4e5ccfff5ab01902
2024-02-01T11:47:08.284816Z 0 [Note] [MY-010949] [Server] Basedir set to /usr/.
2024-02-01T11:47:08.284825Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.33-25) starting as process 1
2024-02-01T11:47:08.378707Z 0 [Note] [MY-012366] [InnoDB] Using Linux native AIO
2024-02-01T11:47:08.378874Z 0 [Note] [MY-010747] [Server] Plugin 'FEDERATED' is disabled.
2024-02-01T11:47:08.378969Z 0 [Note] [MY-010747] [Server] Plugin 'ndbcluster' is disabled.
2024-02-01T11:47:08.378995Z 0 [Note] [MY-010747] [Server] Plugin 'ndbinfo' is disabled.
2024-02-01T11:47:08.379008Z 0 [Note] [MY-010747] [Server] Plugin 'ndb_transid_mysql_connection_map' is disabled.
2024-02-01T11:47:08.380606Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2024-02-01T11:47:08.380650Z 1 [Note] [MY-013546] [InnoDB] Atomic write enabled
2024-02-01T11:47:08.380682Z 1 [Note] [MY-012932] [InnoDB] PUNCH HOLE support available
2024-02-01T11:47:08.380699Z 1 [Note] [MY-012944] [InnoDB] Uses event mutexes
2024-02-01T11:47:08.380704Z 1 [Note] [MY-012945] [InnoDB] GCC builtin __atomic_thread_fence() is used for memory barrier
2024-02-01T11:47:08.380709Z 1 [Note] [MY-012948] [InnoDB] Compressed tables use zlib 1.2.13
2024-02-01T11:47:08.383910Z 1 [Note] [MY-012951] [InnoDB] Using hardware accelerated crc32 and polynomial multiplication.
2024-02-01T11:47:08.384649Z 1 [Note] [MY-012203] [InnoDB] Directories to scan './'
2024-02-01T11:47:08.384713Z 1 [Note] [MY-012204] [InnoDB] Scanning './'
2024-02-01T11:47:08.475498Z 1 [Note] [MY-012208] [InnoDB] Completed space ID check of 4 files.
2024-02-01T11:47:08.476478Z 1 [Note] [MY-012955] [InnoDB] Initializing buffer pool, total size = 1.000000G, instances = 8, chunk size =128.000000M
2024-02-01T11:47:09.777629Z 1 [Note] [MY-012957] [InnoDB] Completed initialization of buffer pool
2024-02-01T11:47:09.875519Z 0 [Note] [MY-011952] [InnoDB] If the mysqld execution user is authorized, page cleaner and LRU manager thread priority can be changed. See the man page of setpriority().
2024-02-01T11:47:09.876490Z 1 [Note] [MY-013532] [InnoDB] Using './#ib_16384_0.dblwr' for doublewrite
2024-02-01T11:47:09.876672Z 1 [Note] [MY-013532] [InnoDB] Using './#ib_16384_1.dblwr' for doublewrite
2024-02-01T11:47:09.878909Z 1 [Note] [MY-013532] [InnoDB] Using './#ib_16384_2.dblwr' for doublewrite
2024-02-01T11:47:09.879071Z 1 [Note] [MY-013532] [InnoDB] Using './#ib_16384_3.dblwr' for doublewrite
2024-02-01T11:47:09.881016Z 1 [Note] [MY-013532] [InnoDB] Using './#ib_16384_4.dblwr' for doublewrite
2024-02-01T11:47:09.881105Z 1 [Note] [MY-013532] [InnoDB] Using './#ib_16384_5.dblwr' for doublewrite
2024-02-01T11:47:09.883223Z 1 [Note] [MY-013532] [InnoDB] Using './#ib_16384_6.dblwr' for doublewrite
2024-02-01T11:47:09.883335Z 1 [Note] [MY-013532] [InnoDB] Using './#ib_16384_7.dblwr' for doublewrite
2024-02-01T11:47:09.885374Z 1 [Note] [MY-013532] [InnoDB] Using './#ib_16384_8.dblwr' for doublewrite
2024-02-01T11:47:09.885460Z 1 [Note] [MY-013532] [InnoDB] Using './#ib_16384_9.dblwr' for doublewrite
2024-02-01T11:47:09.976715Z 1 [Note] [MY-013532] [InnoDB] Using './#ib_16384_10.dblwr' for doublewrite
2024-02-01T11:47:09.976851Z 1 [Note] [MY-013532] [InnoDB] Using './#ib_16384_11.dblwr' for doublewrite
2024-02-01T11:47:09.978925Z 1 [Note] [MY-013532] [InnoDB] Using './#ib_16384_12.dblwr' for doublewrite
2024-02-01T11:47:09.979023Z 1 [Note] [MY-013532] [InnoDB] Using './#ib_16384_13.dblwr' for doublewrite
2024-02-01T11:47:09.981104Z 1 [Note] [MY-013532] [InnoDB] Using './#ib_16384_14.dblwr' for doublewrite
2024-02-01T11:47:09.981195Z 1 [Note] [MY-013532] [InnoDB] Using './#ib_16384_15.dblwr' for doublewrite
2024-02-01T11:47:09.983706Z 1 [Note] [MY-013566] [InnoDB] Double write buffer files: 16
2024-02-01T11:47:09.983744Z 1 [Note] [MY-013565] [InnoDB] Double write buffer pages per instance: 4
2024-02-01T11:47:09.983767Z 1 [Note] [MY-013532] [InnoDB] Using './#ib_16384_0.dblwr' for doublewrite
2024-02-01T11:47:09.983791Z 1 [Note] [MY-013532] [InnoDB] Using './#ib_16384_1.dblwr' for doublewrite
2024-02-01T11:47:09.983814Z 1 [Note] [MY-013532] [InnoDB] Using './#ib_16384_2.dblwr' for doublewrite
2024-02-01T11:47:09.983837Z 1 [Note] [MY-013532] [InnoDB] Using './#ib_16384_3.dblwr' for doublewrite
2024-02-01T11:47:09.983860Z 1 [Note] [MY-013532] [InnoDB] Using './#ib_16384_4.dblwr' for doublewrite
2024-02-01T11:47:09.983905Z 1 [Note] [MY-013532] [InnoDB] Using './#ib_16384_5.dblwr' for doublewrite
2024-02-01T11:47:09.983941Z 1 [Note] [MY-013532] [InnoDB] Using './#ib_16384_6.dblwr' for doublewrite
2024-02-01T11:47:09.983966Z 1 [Note] [MY-013532] [InnoDB] Using './#ib_16384_7.dblwr' for doublewrite
2024-02-01T11:47:09.983995Z 1 [Note] [MY-013532] [InnoDB] Using './#ib_16384_8.dblwr' for doublewrite
2024-02-01T11:47:09.984024Z 1 [Note] [MY-013532] [InnoDB] Using './#ib_16384_9.dblwr' for doublewrite
2024-02-01T11:47:09.984053Z 1 [Note] [MY-013532] [InnoDB] Using './#ib_16384_10.dblwr' for doublewrite
2024-02-01T11:47:09.984085Z 1 [Note] [MY-013532] [InnoDB] Using './#ib_16384_11.dblwr' for doublewrite
2024-02-01T11:47:09.984114Z 1 [Note] [MY-013532] [InnoDB] Using './#ib_16384_12.dblwr' for doublewrite
2024-02-01T11:47:09.984142Z 1 [Note] [MY-013532] [InnoDB] Using './#ib_16384_13.dblwr' for doublewrite
2024-02-01T11:47:09.984168Z 1 [Note] [MY-013532] [InnoDB] Using './#ib_16384_14.dblwr' for doublewrite
2024-02-01T11:47:09.984197Z 1 [Note] [MY-013532] [InnoDB] Using './#ib_16384_15.dblwr' for doublewrite
2024-02-01T11:47:10.375302Z 1 [Note] [MY-013883] [InnoDB] The latest found checkpoint is at lsn = 31379046 in redo log file ./#innodb_redo/#ib_redo9.
2024-02-01T11:47:10.375505Z 1 [Note] [MY-013086] [InnoDB] Starting to parse redo log at lsn = 31378982, whereas checkpoint_lsn = 31379046 and start_lsn = 31378944
2024-02-01T11:47:10.383313Z 1 [Note] [MY-013083] [InnoDB] Log background threads are being started...
2024-02-01T11:47:10.384228Z 1 [Note] [MY-012532] [InnoDB] Applying a batch of 0 redo log records ...
2024-02-01T11:47:10.384258Z 1 [Note] [MY-012535] [InnoDB] Apply batch completed!
2024-02-01T11:47:10.384531Z 1 [Note] [MY-013252] [InnoDB] Using undo tablespace './undo_001'.
2024-02-01T11:47:10.385298Z 1 [Note] [MY-013252] [InnoDB] Using undo tablespace './undo_002'.
2024-02-01T11:47:10.386298Z 1 [Note] [MY-012910] [InnoDB] Opened 2 existing undo tablespaces.
2024-02-01T11:47:10.386367Z 1 [Note] [MY-011980] [InnoDB] GTID recovery trx_no: 1616
2024-02-01T11:47:10.575453Z 1 [Note] [MY-013776] [InnoDB] Parallel initialization of rseg complete
2024-02-01T11:47:10.575555Z 1 [Note] [MY-013777] [InnoDB] Time taken to initialize rseg using 4 thread: 189191 ms.
2024-02-01T11:47:10.575709Z 1 [Note] [MY-012923] [InnoDB] Creating shared tablespace for temporary tables
2024-02-01T11:47:10.575826Z 1 [Note] [MY-012265] [InnoDB] Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2024-02-01T11:47:23.790049Z 1 [Note] [MY-012266] [InnoDB] File './ibtmp1' size is now 12 MB.
2024-02-01T11:47:23.790222Z 1 [Note] [MY-013627] [InnoDB] Scanning temp tablespace dir:'./#innodb_temp/'
2024-02-01T11:47:24.095781Z 1 [Note] [MY-013018] [InnoDB] Created 128 and tracked 128 new rollback segment(s) in the temporary tablespace. 128 are now active.
2024-02-01T11:47:24.096132Z 0 [Note] [MY-011953] [InnoDB] Page cleaner took 14220ms to flush 0 pages
2024-02-01T11:47:24.096174Z 1 [Note] [MY-012976] [InnoDB] Percona XtraDB (http://www.percona.com) 8.0.33-25 started; log sequence number 31379056
2024-02-01T11:47:24.097110Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2024-02-01T11:47:24.104763Z 1 [Note] [MY-011089] [Server] Data dictionary restarting version '80023'.
2024-02-01T11:47:24.875798Z 1 [Note] [MY-012357] [InnoDB] Reading DD tablespace files
2024-02-01T11:47:24.876596Z 1 [Note] [MY-012356] [InnoDB] Scanned 6 tablespaces. Validated 6.
2024-02-01T11:47:24.978145Z 1 [Note] [MY-010006] [Server] Using data dictionary with version '80023'.
2024-02-01T11:47:25.077640Z 0 [Note] [MY-011332] [Server] Plugin mysqlx reported: 'IPv6 is available'
2024-02-01T11:47:25.079339Z 0 [Note] [MY-011323] [Server] Plugin mysqlx reported: 'X Plugin ready for connections. bind-address: '::' port: 33060'
2024-02-01T11:47:25.079379Z 0 [Note] [MY-011323] [Server] Plugin mysqlx reported: 'X Plugin ready for connections. socket: '/var/lib/mysql/mysqlx.sock''
2024-02-01T11:47:25.083094Z 0 [ERROR] [MY-011685] [Repl] Plugin group_replication reported: 'The group_replication_group_name option is mandatory'
2024-02-01T11:47:25.083122Z 0 [ERROR] [MY-011660] [Repl] Plugin group_replication reported: 'Unable to start Group Replication on boot'
2024-02-01T11:47:25.285588Z 0 [Note] [MY-010902] [Server] Thread priority attribute setting in Resource Group SQL shall be ignored due to unsupported platform or insufficient privilege.
2024-02-01T11:47:25.381638Z 0 [Note] [MY-010856] [Server] Failed to open the crashed binlog file when source server is recovering it.
2024-02-01T11:47:25.389606Z 0 [Note] [MY-013911] [Server] Crash recovery finished in binlog engine. No attempts to commit, rollback or prepare any transactions.
2024-02-01T11:47:25.389641Z 0 [Note] [MY-013911] [Server] Crash recovery finished in InnoDB engine. No attempts to commit, rollback or prepare any transactions.
2024-02-01T11:47:25.477849Z 0 [Note] [MY-012487] [InnoDB] DDL log recovery : begin
2024-02-01T11:47:25.477922Z 0 [Note] [MY-012488] [InnoDB] DDL log recovery : end
2024-02-01T11:47:25.478263Z 0 [Note] [MY-011946] [InnoDB] Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
2024-02-01T11:47:25.576569Z 0 [Note] [MY-011946] [InnoDB] Buffer pool(s) load completed at 240201 17:17:25
2024-02-01T11:47:25.907428Z 0 [Note] [MY-010303] [Server] Skipping generation of SSL certificates as options related to SSL are specified.
2024-02-01T11:47:25.911011Z 0 [Warning] [MY-010068] [Server] CA certificate /etc/mysql/mysql-tls-secret/ca.crt is self signed.
2024-02-01T11:47:25.911056Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel.
2024-02-01T11:47:26.675856Z 0 [Warning] [MY-013595] [Server] Failed to initialize TLS for channel: mysql_admin. See below for the description of exact issue.
2024-02-01T11:47:26.675936Z 0 [Warning] [MY-010069] [Server] Failed to set up SSL because of the following SSL library error: SSL context is not usable without certificate and private key
2024-02-01T11:47:26.675960Z 0 [System] [MY-013603] [Server] No TLS configuration was given for channel mysql_admin; re-using TLS configuration of channel mysql_main.
2024-02-01T11:47:26.677174Z 0 [Note] [MY-010308] [Server] Skipping generation of RSA key pair through --sha256_password_auto_generate_rsa_keys as key files are present in data directory.
2024-02-01T11:47:26.678237Z 0 [Note] [MY-010308] [Server] Skipping generation of RSA key pair through --caching_sha2_password_auto_generate_rsa_keys as key files are present in data directory.
2024-02-01T11:47:26.681964Z 0 [Note] [MY-010252] [Server] Server hostname (bind-address): '10.49.203.6'; port: 33062
2024-02-01T11:47:26.682003Z 0 [Note] [MY-010264] [Server] - '10.49.203.6' resolves to '10.49.203.6';
2024-02-01T11:47:26.682023Z 0 [Note] [MY-010251] [Server] Server socket created on IP: '10.49.203.6'.
2024-02-01T11:47:26.682069Z 0 [Note] [MY-010252] [Server] Server hostname (bind-address): '*'; port: 3306
2024-02-01T11:47:26.682105Z 0 [Note] [MY-010253] [Server] IPv6 is available.
2024-02-01T11:47:26.682115Z 0 [Note] [MY-010264] [Server] - '::' resolves to '::';
2024-02-01T11:47:26.682130Z 0 [Note] [MY-010251] [Server] Server socket created on IP: '::'.
2024-02-01T11:47:26.876712Z 0 [Note] [MY-011025] [Repl] Failed to start replica threads for channel ''.
2024-02-01T11:47:26.878410Z 7 [Note] [MY-010051] [Server] Event Scheduler: scheduler thread started with id 7
2024-02-01T11:47:26.878514Z 0 [Note] [MY-011240] [Server] Plugin mysqlx reported: 'Using SSL configuration from MySQL Server'
2024-02-01T11:47:26.882241Z 0 [Note] [MY-011243] [Server] Plugin mysqlx reported: 'Using OpenSSL for TLS connections'
2024-02-01T11:47:26.882404Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: '::' port: 33060, socket: /var/lib/mysql/mysqlx.sock
2024-02-01T11:47:26.882524Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.33-25' socket: '/var/lib/mysql/mysql.sock' port: 3306 Percona Server (GPL), Release 25, Revision 60c9e2c5.
2024-02-01T11:47:26.882545Z 0 [System] [MY-013292] [Server] Admin interface ready for connections, address: '10.49.203.6' port: 33062
2024-02-01T11:47:26.882618Z 0 [System] [MY-013172] [Server] Received SHUTDOWN from user <via user signal>. Shutting down mysqld (Version: 8.0.33-25).
2024-02-01T11:47:26.975723Z 0 [Note] [MY-010067] [Server] Giving 3 client threads a chance to die gracefully
2024-02-01T11:47:26.975769Z 0 [Note] [MY-010117] [Server] Shutting down replica threads
2024-02-01T11:47:26.975778Z 0 [Note] [MY-010054] [Server] Event Scheduler: Killing the scheduler thread, thread id 7
2024-02-01T11:47:26.975790Z 0 [Note] [MY-010050] [Server] Event Scheduler: Waiting for the scheduler thread to reply
2024-02-01T11:47:26.975866Z 0 [Note] [MY-010048] [Server] Event Scheduler: Stopped
2024-02-01T11:47:28.975962Z 0 [Note] [MY-010118] [Server] Forcefully disconnecting 2 remaining clients
2024-02-01T11:47:28.976435Z 0 [Note] [MY-010043] [Server] Event Scheduler: Purging the queue. 0 events
2024-02-01T11:47:28.977734Z 0 [Note] [MY-012330] [InnoDB] FTS optimize thread exiting.
2024-02-01T11:47:29.880543Z 0 [Note] [MY-010120] [Server] Binlog end
2024-02-01T11:47:29.882818Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'clone'
2024-02-01T11:47:29.882864Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'group_replication'
2024-02-01T11:47:29.882919Z 0 [Note] [MY-011665] [Repl] Plugin group_replication reported: 'All Group Replication server observers have been successfully unregistered'
I can reproduce it when I set
containerSecurityContext:
runAsUser: 999
Can you tell me more about your environment, why did you pick 999? Our container image uses UID 1001 for mysql user. So setting runAsUser: 1001 would work. Will it work for you?
This worked. Thanks a lot for the image Dockerfile info.
Though i find the percona-server ERROR logs didn’t shown any permission related issue. Can this be handled in percona-server?
I think we can improve it. Will look into it.
Glad that it worked for you!
1 Like