# Percona XtraDB Cluster Operator: /var/lib/mysql has wrong permissions **URL:** \ **Category:** Percona Operator for MySQL\ **Created:** [June 21, 2019, 6:25am UTC](https://forums.percona.com/t/percona-xtradb-cluster-operator-var-lib-mysql-has-wrong-permissions/7072 "2019-06-21T06:25:33Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![melnik](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/melnik/32/1268_2.png) [@melnik](https://forums.percona.com/u/melnik)\ **Post date:** [June 21, 2019, 6:25am UTC](https://forums.percona.com/t/percona-xtradb-cluster-operator-var-lib-mysql-has-wrong-permissions/7072/1 "2019-06-21T06:25:33Z")
Hello, I’m trying to deploy Percona XtraDB Cluster Operator in a k8s cluster according to the manual that is being exposed here: [URL=“[Install Percona XtraDB Cluster on Kubernetes](https://www.percona.com/doc/kubernetes-operator-for-pxc/kubernetes.html)”][https://www.percona.com/doc/kubernet...ubernetes.html[/URL]](https://www.percona.com/doc/kubernet...ubernetes.html%5B/URL%5D). After performing all necessary steps I see that the `cluster1-pxc-0` is being constantly restarted: ```auto NAME READY STATUS RESTARTS AGE cluster1-proxysql-0 3/3 Running 0 37m cluster1-proxysql-1 3/3 Running 0 37m cluster1-proxysql-2 3/3 Running 0 37m cluster1-pxc-0 0/1 CrashLoopBackOff 10 37m percona-xtradb-cluster-operator-6bc6889544-4l5fb 1/1 Running 0 40m ``` The pod’s log shows that it’s having a problem accessing to `/var/lib/mysql` directory which is mounted to a GlusterFS storage: ```auto ++ id -u + USER_ID=1001 + _MYSQL_ROOT_HOST=% + '[' '' = - ']' + echo 'Percona XtraDB Cluster: Finding peers' + PXC_SERVICE=cluster1-pxc-unready + echo 'Using service name: cluster1-pxc-unready' + /usr/bin/peer-list -on-start=/usr/bin/configure-pxc.sh -service=cluster1-pxc-unready Percona XtraDB Cluster: Finding peers Using service name: cluster1-pxc-unready 2019/06/21 11:38:45 Peer finder enter 2019/06/21 11:38:45 Determined Domain to be pxc.svc.k8s. ***.***.*** 2019/06/21 11:38:45 Peer list updated was [] now [10-233-96-46.cluster1-pxc-unready.pxc.svc.k8s. ***.***.***] 2019/06/21 11:38:45 execing: /usr/bin/configure-pxc.sh with stdin: 10-233-96-46.cluster1-pxc-unready.pxc.svc.k8s. ***.***.*** 2019/06/21 11:38:45 read line 10-233-96-46.cluster1-pxc-unready.pxc.svc.k8s. ***.***.*** 2019/06/21 11:38:46 Peer finder exiting ++ mysqld --verbose --wsrep_provider= --help ++ awk '$1 == "datadir" { print $2; exit }' + DATADIR=/var/lib/mysql/ + '[' -z '' ']' + DATADIR=/var/lib/mysql + cat /etc/mysql/node.cnf [mysqld] pxc-encrypt-cluster-traffic=ON ssl-ca=/etc/mysql/ssl-internal/ca.crt ssl-key=/etc/mysql/ssl-internal/tls.key ssl-cert=/etc/mysql/ssl-internal/tls.crt ignore-db-dir=lost+found datadir=/var/lib/mysql socket=/tmp/mysql.sock server_id=10 binlog_format=ROW default_storage_engine=InnoDB innodb_flush_log_at_trx_commit = 0 innodb_flush_method = O_DIRECT innodb_file_per_table = 1 innodb_autoinc_lock_mode=2 bind_address = 0.0.0.0 wsrep_slave_threads=2 wsrep_cluster_address=gcomm:// wsrep_node_address=10.233.96.46 wsrep_provider=/usr/lib64/galera3/libgalera_smm.so wsrep_cluster_name=cluster1-pxc wsrep_sst_method=xtrabackup-v2 wsrep_sst_auth='xtrabackup:rpt7tM9CvXRL3' [client] socket=/tmp/mysql.sock ++ grep wsrep_cluster_address /etc/mysql/node.cnf ++ sed -e 's^.*gcomm://^^' Cluster address set to: + WSREP_CLUSTER_ADDRESS= + echo 'Cluster address set to: ' Cluster address is empty! + '[' -z '' ']' + echo 'Cluster address is empty! ' + '[' '!' -z '' ']' + '[' '!' -e /var/lib/mysql/mysql ']' Running with password ::VMx7s4rHgdThp:: + echo 'Running with password ::VMx7s4rHgdThp::' + '[' -z VMx7s4rHgdThp -a -z '' -a -z '' -a -z '' ']' + '[' '!' -z '' -a -z VMx7s4rHgdThp ']' + rm -rf '/var/lib/mysql/*' + mkdir -p /var/lib/mysql + echo 'Running --initialize-insecure on /var/lib/mysql' + mysqld --initialize-insecure --skip-ssl Running --initialize-insecure on /var/lib/mysql mysqld: Can't create/write to file '/var/lib/mysql/is_writable' (Errcode: 13 - Permission denied) 2019-06-21T11:38:52.200250Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details). 2019-06-21T11:38:52.200707Z 0 [Warning] WSREP: Node is running in bootstrap/initialize mode. Disabling pxc_strict_mode checks 2019-06-21T11:38:52.223799Z 0 [ERROR] --initialize specified but the data directory exists and is not writable. Aborting. 2019-06-21T11:38:52.223875Z 0 [ERROR] Aborting + echo 'Finished --initialize-insecure' + pid=39 + mysql=(mysql --protocol=socket -uroot) + mysqld --user=mysql --datadir=/var/lib/mysql --skip-networking + for i in '{30..0}' Finished --initialize-insecure + mysql --protocol=socket -uroot + echo 'SELECT 1' + echo 'MySQL init process in progress...' + sleep 1 MySQL init process in progress... 2019-06-21T11:38:52.527091Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details). 2019-06-21T11:38:52.561605Z 0 [Warning] Can't create test file /var/lib/mysql/cluster1-pxc-0.lower-test 2019-06-21T11:38:52.561840Z 0 [Note] mysqld (mysqld 5.7.25-28-57) starting as process 39 ... 2019-06-21T11:38:52.584493Z 0 [Warning] Can't create test file /var/lib/mysql/cluster1-pxc-0.lower-test 2019-06-21T11:38:52.605591Z 0 [Warning] Can't create test file /var/lib/mysql/cluster1-pxc-0.lower-test 2019-06-21T11:38:52.607954Z 0 [Note] WSREP: Skipping automatic SSL certificate generation (enabled only in bootstrap mode) 2019-06-21T11:38:52.608038Z 0 [Note] WSREP: Setting wsrep_ready to false 2019-06-21T11:38:52.608063Z 0 [Note] WSREP: No pre-stored wsrep-start position found. Skipping position initialization. 2019-06-21T11:38:52.608073Z 0 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib64/galera3/libgalera_smm.so' 2019-06-21T11:38:52.620062Z 0 [Note] WSREP: wsrep_load(): Galera 3.35(rddf9876) by Codership Oy loaded successfully. 2019-06-21T11:38:52.620601Z 0 [Note] WSREP: CRC-32C: using hardware acceleration. 2019-06-21T11:38:52.628488Z 0 [Warning] WSREP: Could not open state file for reading: '/var/lib/mysql//grastate.dat' 2019-06-21T11:38:52.628540Z 0 [Warning] WSREP: No persistent state found. Bootstraping with default state 2019-06-21T11:38:52.636787Z 0 [ERROR] WSREP: Could not open state file for writing: '/var/lib/mysql//grastate.dat'. Check permissions and/or disk space.: 13 (Permission denied) at galera/src/saved_state.cpp:SavedState():57 2019-06-21T11:38:52.636864Z 0 [ERROR] WSREP: Failed to initialize wsrep_provider (reason:7). Must shutdown 2019-06-21T11:38:52.636891Z 0 [ERROR] Aborting 2019-06-21T11:38:52.636932Z 0 [Note] Binlog end 2019-06-21T11:38:52.640434Z 0 [Note] mysqld: Shutdown complete + for i in '{30..0}' + echo 'SELECT 1' + mysql --protocol=socket -uroot + echo 'MySQL init process in progress...' + sleep 1 MySQL init process in progress... ***SKIPPED*** + '[' 0 = 0 ']' + echo 'MySQL init process failed.' MySQL init process failed. + exit 1 ``` I’m trying to figure out, why doesn’t the process have the permissions to write to write to /var/lib/mysql. Here’s what I see when I run ls -al /var/lib/mysql in the container: ```auto total 8 drwxr-xr-x. 2 root root 4096 Jun 21 10:32 . drwxr-xr-x. 17 root root 4096 May 10 16:57 .. ``` The directory belongs to root:root, not to mysql (UID=1001), that’s why the initialization script couldn’t initialize the database properly. Shouldn’t the directory belong to mysql? Why doesn’t it? Have I done something wrong? Here’s what I see when I run mount in the container: ```auto ***SKIPPED*** 10.13.1.16:storage1/pxc/datadir-cluster1-pxc-0 on /var/lib/mysql type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072) ***SKIPPED*** ``` My PVC looks like that: ```auto apiVersion: v1 kind: PersistentVolumeClaim metadata: annotations: pv.kubernetes.io/bind-completed: "yes" pv.kubernetes.io/bound-by-controller: "yes" creationTimestamp: "2019-06-21T10:56:46Z" finalizers: - kubernetes.io/pvc-protection labels: app.kubernetes.io/component: pxc app.kubernetes.io/instance: cluster1 app.kubernetes.io/managed-by: percona-xtradb-cluster-operator app.kubernetes.io/name: percona-xtradb-cluster app.kubernetes.io/part-of: percona-xtradb-cluster name: datadir-cluster1-pxc-0 namespace: pxc resourceVersion: "3024217" selfLink: /api/v1/namespaces/pxc/persistentvolumeclaims/datadir-cluster1-pxc-0 uid: 4308b987-9413-11e9-ad60-02001a7f0008 spec: accessModes: - ReadWriteOnce resources: requests: storage: 6Gi volumeMode: Filesystem volumeName: datadir-cluster1-pxc-0 status: accessModes: - ReadWriteOnce capacity: storage: 6Gi phase: Bound ``` The PV looks like that: ```auto apiVersion: v1 kind: PersistentVolume metadata: annotations: kubectl.kubernetes.io/last-applied-configuration: | {"apiVersion":"v1","kind":"PersistentVolume","metadata":{"annotations":{},"name":"datadir-cluster1-pxc-0"},"spec":{"accessModes":["ReadWriteOnce"],"capacity":{"storage":"6Gi"},"claimRef":{"name":"datadir-cluster1-pxc-0","namespace":"pxc"},"glusterfs":{"endpoints":"glusterfs-cluster","path":"storage1/pxc/datadir-cluster1-pxc-0","readOnly":false},"persistentVolumeReclaimPolicy":"Retain"}} creationTimestamp: "2019-06-21T10:57:30Z" finalizers: - kubernetes.io/pv-protection name: datadir-cluster1-pxc-0 resourceVersion: "3024215" selfLink: /api/v1/persistentvolumes/datadir-cluster1-pxc-0 uid: 5d97bbd4-9413-11e9-ad60-02001a7f0008 spec: accessModes: - ReadWriteOnce capacity: storage: 6Gi claimRef: apiVersion: v1 kind: PersistentVolumeClaim name: datadir-cluster1-pxc-0 namespace: pxc resourceVersion: "3023964" uid: 4308b987-9413-11e9-ad60-02001a7f0008 glusterfs: endpoints: glusterfs-cluster path: storage1/pxc/datadir-cluster1-pxc-0 persistentVolumeReclaimPolicy: Retain volumeMode: Filesystem status: phase: Bound ``` Thanks in advance for any tips and clues. ---
**Author:** ![libardolara](https://avatars.discourse-cdn.com/v4/letter/l/a698b9/32.png) [@libardolara](https://forums.percona.com/u/libardolara)\ **Post date:** [April 30, 2020, 9:09am UTC](https://forums.percona.com/t/percona-xtradb-cluster-operator-var-lib-mysql-has-wrong-permissions/7072/2 "2020-04-30T09:09:04Z")
Hi Melnik, I found the same issue and realized that in some k8s implementation non-root users do not have the privilege to modify the pvc mount dir as they are not owners. The solution would be to have an InitContainers to run before with a command to assign ownership to the user 1001 (That is the user specified in the Dockerfile).  ```auto initContainers: - name: "permissionsfix" image: "busybox:1.25.0" imagePullPolicy: IfNotPresent command: ["/bin/sh", "-c"] args: - chown 1001:1001 /var/lib/mysql; volumeMounts: - name: datadir mountPath: /var/lib/mysql ``` The only problem is that Percona Operator doesn’t support this configuration, at least I haven’t found a way to do it with the current operator. ---
**Author:** ![Ivan\_Pylypenko](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/ivan_pylypenko/32/2452_2.png) [@Ivan\_Pylypenko](https://forums.percona.com/u/Ivan_Pylypenko)\ **Post date:** [April 30, 2020, 11:07am UTC](https://forums.percona.com/t/percona-xtradb-cluster-operator-var-lib-mysql-has-wrong-permissions/7072/3 "2020-04-30T11:07:35Z")
Hello If something like this happen please consider securityContext related field in PXC custom resource. Kubernetes shall adjust permissions properly in that case. ---
**Author:** ![Brad\_Baker1](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/brad_baker1/32/6594_2.png) [@Brad\_Baker1](https://forums.percona.com/u/Brad_Baker1)\ **Post date:** [September 9, 2020, 4:34pm UTC](https://forums.percona.com/t/percona-xtradb-cluster-operator-var-lib-mysql-has-wrong-permissions/7072/4 "2020-09-09T16:34:52Z")
@IvanPylypenko  I am having the same issue trying to deploy percona xtradb cluster to my kubernetes instances using helm. I have used the following:securityContext:   runAsUser: 1001   runAsGroup: 1001   fsGroup: 1001 And then I ran chmod 777 and chown 1001:1001 on my Persistent Volume HostPath. Yet I still get this error… I’m scratching my head. Can you shed any light on how to get this to work?