Percona XtraDB Cluster Operator: /var/lib/mysql has wrong permissions

melnik · June 21, 2019, 6:25am

Hello,

I’m trying to deploy Percona XtraDB Cluster Operator in a k8s cluster according to the manual that is being exposed here: [URL=“Install Percona XtraDB Cluster on Kubernetes”]https://www.percona.com/doc/kubernet...ubernetes.html[/URL].

After performing all necessary steps I see that the cluster1-pxc-0 is being constantly restarted:

NAME READY STATUS RESTARTS AGE
cluster1-proxysql-0 3/3 Running 0 37m
cluster1-proxysql-1 3/3 Running 0 37m
cluster1-proxysql-2 3/3 Running 0 37m
cluster1-pxc-0 0/1 CrashLoopBackOff 10 37m
percona-xtradb-cluster-operator-6bc6889544-4l5fb 1/1 Running 0 40m

The pod’s log shows that it’s having a problem accessing to /var/lib/mysql directory which is mounted to a GlusterFS storage:

++ id -u
+ USER_ID=1001
+ _MYSQL_ROOT_HOST=%
+ '[' '' = - ']'
+ echo 'Percona XtraDB Cluster: Finding peers'
+ PXC_SERVICE=cluster1-pxc-unready
+ echo 'Using service name: cluster1-pxc-unready'
+ /usr/bin/peer-list -on-start=/usr/bin/configure-pxc.sh -service=cluster1-pxc-unready
Percona XtraDB Cluster: Finding peers
Using service name: cluster1-pxc-unready
2019/06/21 11:38:45 Peer finder enter
2019/06/21 11:38:45 Determined Domain to be pxc.svc.k8s.***.***.***
2019/06/21 11:38:45 Peer list updated
was []
now [10-233-96-46.cluster1-pxc-unready.pxc.svc.k8s.***.***.***]
2019/06/21 11:38:45 execing: /usr/bin/configure-pxc.sh with stdin: 10-233-96-46.cluster1-pxc-unready.pxc.svc.k8s.***.***.***
2019/06/21 11:38:45 read line 10-233-96-46.cluster1-pxc-unready.pxc.svc.k8s.***.***.***
2019/06/21 11:38:46 Peer finder exiting
++ mysqld --verbose --wsrep_provider= --help
++ awk '$1 == "datadir" { print $2; exit }'
+ DATADIR=/var/lib/mysql/
+ '[' -z '' ']'
+ DATADIR=/var/lib/mysql
+ cat /etc/mysql/node.cnf
[mysqld]
pxc-encrypt-cluster-traffic=ON
ssl-ca=/etc/mysql/ssl-internal/ca.crt
ssl-key=/etc/mysql/ssl-internal/tls.key
ssl-cert=/etc/mysql/ssl-internal/tls.crt

ignore-db-dir=lost+found
datadir=/var/lib/mysql
socket=/tmp/mysql.sock

server_id=10
binlog_format=ROW
default_storage_engine=InnoDB

innodb_flush_log_at_trx_commit = 0
innodb_flush_method = O_DIRECT
innodb_file_per_table = 1
innodb_autoinc_lock_mode=2

bind_address = 0.0.0.0

wsrep_slave_threads=2
wsrep_cluster_address=gcomm://

wsrep_node_address=10.233.96.46

wsrep_provider=/usr/lib64/galera3/libgalera_smm.so

wsrep_cluster_name=cluster1-pxc

wsrep_sst_method=xtrabackup-v2
wsrep_sst_auth='xtrabackup:rpt7tM9CvXRL3'

[client]
socket=/tmp/mysql.sock
++ grep wsrep_cluster_address /etc/mysql/node.cnf
++ sed -e 's^.*gcomm://^^'
Cluster address set to:
+ WSREP_CLUSTER_ADDRESS=
+ echo 'Cluster address set to: '
Cluster address is empty!
+ '[' -z '' ']'
+ echo 'Cluster address is empty! '
+ '[' '!' -z '' ']'
+ '[' '!' -e /var/lib/mysql/mysql ']'
Running with password ::VMx7s4rHgdThp::
+ echo 'Running with password ::VMx7s4rHgdThp::'
+ '[' -z VMx7s4rHgdThp -a -z '' -a -z '' -a -z '' ']'
+ '[' '!' -z '' -a -z VMx7s4rHgdThp ']'
+ rm -rf '/var/lib/mysql/*'
+ mkdir -p /var/lib/mysql
+ echo 'Running --initialize-insecure on /var/lib/mysql'
+ mysqld --initialize-insecure --skip-ssl
Running --initialize-insecure on /var/lib/mysql
mysqld: Can't create/write to file '/var/lib/mysql/is_writable' (Errcode: 13 - Permission denied)
2019-06-21T11:38:52.200250Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2019-06-21T11:38:52.200707Z 0 [Warning] WSREP: Node is running in bootstrap/initialize mode. Disabling pxc_strict_mode checks
2019-06-21T11:38:52.223799Z 0 [ERROR] --initialize specified but the data directory exists and is not writable. Aborting.
2019-06-21T11:38:52.223875Z 0 [ERROR] Aborting

+ echo 'Finished --initialize-insecure'
+ pid=39
+ mysql=(mysql --protocol=socket -uroot)
+ mysqld --user=mysql --datadir=/var/lib/mysql --skip-networking
+ for i in '{30..0}'
Finished --initialize-insecure
+ mysql --protocol=socket -uroot
+ echo 'SELECT 1'
+ echo 'MySQL init process in progress...'
+ sleep 1
MySQL init process in progress...
2019-06-21T11:38:52.527091Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2019-06-21T11:38:52.561605Z 0 [Warning] Can't create test file /var/lib/mysql/cluster1-pxc-0.lower-test
2019-06-21T11:38:52.561840Z 0 [Note] mysqld (mysqld 5.7.25-28-57) starting as process 39 ...
2019-06-21T11:38:52.584493Z 0 [Warning] Can't create test file /var/lib/mysql/cluster1-pxc-0.lower-test
2019-06-21T11:38:52.605591Z 0 [Warning] Can't create test file /var/lib/mysql/cluster1-pxc-0.lower-test
2019-06-21T11:38:52.607954Z 0 [Note] WSREP: Skipping automatic SSL certificate generation (enabled only in bootstrap mode)
2019-06-21T11:38:52.608038Z 0 [Note] WSREP: Setting wsrep_ready to false
2019-06-21T11:38:52.608063Z 0 [Note] WSREP: No pre-stored wsrep-start position found. Skipping position initialization.
2019-06-21T11:38:52.608073Z 0 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib64/galera3/libgalera_smm.so'
2019-06-21T11:38:52.620062Z 0 [Note] WSREP: wsrep_load(): Galera 3.35(rddf9876) by Codership Oy <info&#64;codership.com> loaded successfully.
2019-06-21T11:38:52.620601Z 0 [Note] WSREP: CRC-32C: using hardware acceleration.
2019-06-21T11:38:52.628488Z 0 [Warning] WSREP: Could not open state file for reading: '/var/lib/mysql//grastate.dat'
2019-06-21T11:38:52.628540Z 0 [Warning] WSREP: No persistent state found. Bootstraping with default state
2019-06-21T11:38:52.636787Z 0 [ERROR] WSREP: Could not open state file for writing: '/var/lib/mysql//grastate.dat'. Check permissions and/or disk space.: 13 (Permission denied)
at galera/src/saved_state.cpp:SavedState():57
2019-06-21T11:38:52.636864Z 0 [ERROR] WSREP: Failed to initialize wsrep_provider (reason:7). Must shutdown
2019-06-21T11:38:52.636891Z 0 [ERROR] Aborting

2019-06-21T11:38:52.636932Z 0 [Note] Binlog end
2019-06-21T11:38:52.640434Z 0 [Note] mysqld: Shutdown complete
+ for i in '{30..0}'
+ echo 'SELECT 1'
+ mysql --protocol=socket -uroot
+ echo 'MySQL init process in progress...'
+ sleep 1
MySQL init process in progress...
***SKIPPED***
+ '[' 0 = 0 ']'
+ echo 'MySQL init process failed.'
MySQL init process failed.
+ exit 1

I’m trying to figure out, why doesn’t the process have the permissions to write to write to /var/lib/mysql.

Here’s what I see when I run ls -al /var/lib/mysql in the container:

total 8
drwxr-xr-x. 2 root root 4096 Jun 21 10:32 .
drwxr-xr-x. 17 root root 4096 May 10 16:57 ..

The directory belongs to root:root, not to mysql (UID=1001), that’s why the initialization script couldn’t initialize the database properly.

Shouldn’t the directory belong to mysql? Why doesn’t it? Have I done something wrong?

Here’s what I see when I run mount in the container:

***SKIPPED***
10.13.1.16:storage1/pxc/datadir-cluster1-pxc-0 on /var/lib/mysql type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)
***SKIPPED***

My PVC looks like that:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
annotations:
pv.kubernetes.io/bind-completed: "yes"
pv.kubernetes.io/bound-by-controller: "yes"
creationTimestamp: "2019-06-21T10:56:46Z"
finalizers:
- kubernetes.io/pvc-protection
labels:
app.kubernetes.io/component: pxc
app.kubernetes.io/instance: cluster1
app.kubernetes.io/managed-by: percona-xtradb-cluster-operator
app.kubernetes.io/name: percona-xtradb-cluster
app.kubernetes.io/part-of: percona-xtradb-cluster
name: datadir-cluster1-pxc-0
namespace: pxc
resourceVersion: "3024217"
selfLink: /api/v1/namespaces/pxc/persistentvolumeclaims/datadir-cluster1-pxc-0
uid: 4308b987-9413-11e9-ad60-02001a7f0008
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 6Gi
volumeMode: Filesystem
volumeName: datadir-cluster1-pxc-0
status:
accessModes:
- ReadWriteOnce
capacity:
storage: 6Gi
phase: Bound

The PV looks like that:

apiVersion: v1
kind: PersistentVolume
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"v1","kind":"PersistentVolume","metadata":{"annotations":{},"name":"datadir-cluster1-pxc-0"},"spec":{"accessModes":["ReadWriteOnce"],"capacity":{"storage":"6Gi"},"claimRef":{"name":"datadir-cluster1-pxc-0","namespace":"pxc"},"glusterfs":{"endpoints":"glusterfs-cluster","path":"storage1/pxc/datadir-cluster1-pxc-0","readOnly":false},"persistentVolumeReclaimPolicy":"Retain"}}
creationTimestamp: "2019-06-21T10:57:30Z"
finalizers:
- kubernetes.io/pv-protection
name: datadir-cluster1-pxc-0
resourceVersion: "3024215"
selfLink: /api/v1/persistentvolumes/datadir-cluster1-pxc-0
uid: 5d97bbd4-9413-11e9-ad60-02001a7f0008
spec:
accessModes:
- ReadWriteOnce
capacity:
storage: 6Gi
claimRef:
apiVersion: v1
kind: PersistentVolumeClaim
name: datadir-cluster1-pxc-0
namespace: pxc
resourceVersion: "3023964"
uid: 4308b987-9413-11e9-ad60-02001a7f0008
glusterfs:
endpoints: glusterfs-cluster
path: storage1/pxc/datadir-cluster1-pxc-0
persistentVolumeReclaimPolicy: Retain
volumeMode: Filesystem
status:
phase: Bound

Thanks in advance for any tips and clues.

libardolara · April 30, 2020, 9:09am

Hi Melnik,

I found the same issue and realized that in some k8s implementation non-root users do not have the privilege to modify the pvc mount dir as they are not owners.

The solution would be to have an InitContainers to run before with a command to assign ownership to the user 1001 (That is the user specified in the Dockerfile).

initContainers:
- name: "permissionsfix"
image: "busybox:1.25.0"
imagePullPolicy: IfNotPresent
command: ["/bin/sh", "-c"]
args:
- chown 1001:1001 /var/lib/mysql;
volumeMounts:
- name: datadir
mountPath: /var/lib/mysql

The only problem is that Percona Operator doesn’t support this configuration, at least I haven’t found a way to do it with the current operator.

Ivan_Pylypenko · April 30, 2020, 11:07am

Hello
If something like this happen please consider securityContext related field in PXC custom resource. Kubernetes shall adjust permissions properly in that case.

Brad_Baker1 · September 9, 2020, 4:34pm

@IvanPylypenko I am having the same issue trying to deploy percona xtradb cluster to my kubernetes instances using helm.

I have used the following:securityContext:
runAsUser: 1001
runAsGroup: 1001
fsGroup: 1001
And then I ran chmod 777 and chown 1001:1001 on my Persistent Volume HostPath. Yet I still get this error… I’m scratching my head. Can you shed any light on how to get this to work?

Topic		Replies	Views
Unable to configure Percona MySQL group replication cluster on K8s with version v0.6.0 Percona Operator for MySQL	4	470	February 5, 2024
Percona MySQL operator kubernetes on EKS Percona XtraDB Cluster 8.x community , mysql , percona , new-release	5	1020	March 13, 2022
Issue while installing Percona XtraDB Cluster on Kubernetes Percona Operator for MySQL percona	3	1035	January 15, 2024
Percona Kubernetes Operator for Percona XtraDB Cluster ver 1.5.0 Released Percona Operator for MySQL software-release-not , duplicate-post	0	515	July 21, 2020
Query: Could you please explain "cluster1-proxysql-unready" . Percona Operator for MySQL	2	961	May 15, 2020

Percona XtraDB Cluster Operator: /var/lib/mysql has wrong permissions

Related topics