Simplest defaults seem to not work?

MongoDB Percona Operator for MongoDB
1

Hey,

I’m getting started with this operator and am installing via helm, and creating a test resource from cr.yaml using kubectl apply -f https://raw.githubusercontent.com/percona/percona-server-mongodb-operator/main/deploy/cr.yaml

I’m running under minikube with the kvm2 driver and docker as the container runtime. I’ve also tried containerd, and using the docker driver instead. All have the same result as below.

The result hasn’t quite been what I’d expected:

~/code/nc-helm >>> kubectl get pods                                                                                                                                                                               
NAME                                      READY   STATUS       RESTARTS   AGE
mdb-psmdb-operator-d64bc65fd-spb9t        1/1     Running      0          9m32s
my-cluster-name-cfg-0                     0/2     Init:0/1     0          2m53s
my-cluster-name-mongos-74f9b75bf4-9gfqc   0/1     Init:0/1     0          2m49s
my-cluster-name-mongos-74f9b75bf4-hfk58   0/1     Running      0          2m49s
my-cluster-name-mongos-74f9b75bf4-r2tvw   0/1     Running      0          2m49s
my-cluster-name-rs0-0                     0/2     Init:Error   3          2m51s
~/code/nc-helm >>> kubectl logs my-cluster-name-rs0-0                                                                                                                                                             
error: a container name must be specified for pod my-cluster-name-rs0-0, choose one of: [mongod backup-agent] or one of the init containers: [mongo-init]
~/code/nc-helm >>> kubectl logs my-cluster-name-rs0-0 mongod                                                                                                                                                   [1]
Error from server (BadRequest): container "mongod" in pod "my-cluster-name-rs0-0" is waiting to start: PodInitializing
~/code/nc-helm >>> kubectl logs my-cluster-name-rs0-0 mongo-init                                                                                                                                               [1]
++ id -u
++ id -g
+ install -o 99 -g 99 -m 0755 -D /ps-entry.sh /data/db/ps-entry.sh
install: cannot create regular file '/data/db/ps-entry.sh': Permission denied
~/code/nc-helm >>> kubectl logs my-cluster-name-cfg-0 mongo-init                                                                                                                                               [1]
++ id -u
++ id -g
+ install -o 99 -g 99 -m 0755 -D /ps-entry.sh /data/db/ps-entry.sh
install: cannot create regular file '/data/db/ps-entry.sh': Permission denied

The only container which seems to run by default are the ones running mongos. Is this expected? I’d expect the cr.yaml from master to work with defaults set.

~/code/nc-helm >>> kubectl version                                                                                                                                                                                
Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.3", GitCommit:"ca643a4d1f7bfe34773c74f79527be4afd95bf39", GitTreeState:"archive", BuildDate:"2021-07-16T17:16:46Z", GoVersion:"go1.16.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.2", GitCommit:"092fbfbf53427de67cac1e9fa54aaa09a28371d7", GitTreeState:"clean", BuildDate:"2021-06-16T12:53:14Z", GoVersion:"go1.16.5", Compiler:"gc", Platform:"linux/amd64"}
1 Like
2

Hi @jhackett,

The operator creates a persistent volume claim and mounts it to /data/db path. It seems there might be a problem in your setup. Could you please share your PersistentVolume config?

1 Like
3

Sure. I’m using minikube with defaults set and the kvm2 driver.

PVCs:

$ kubectl get pvc                                                                                                                                                                                             
NAME                                STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
mongod-data-my-cluster-name-cfg-0   Bound    pvc-cd7e59dd-f38f-4b6d-abf4-db6936f42864   3Gi        RWO            standard       29h
mongod-data-my-cluster-name-cfg-1   Bound    pvc-dd94101a-e752-4b26-ab31-0bc58f32514a   3Gi        RWO            standard       29h
mongod-data-my-cluster-name-rs0-0   Bound    pvc-1b9e2c85-f798-4ef5-ba9f-90567d15720f   3Gi        RWO            standard       29h
mongod-data-my-cluster-name-rs0-1   Bound    pvc-a7ce7077-e63c-4855-a0d5-528dca452fe6   3Gi        RWO            standard       29h

Storage class:

$ kubectl describe storageclass standard                                                                                                                                                                      
Name:            standard
IsDefaultClass:  Yes
Annotations:     kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"storage.k8s.io/v1","kind":"StorageClass","metadata":{"annotations":{"storageclass.kubernetes.io/is-default-class":"true"},"labels":{"addonmanager.kubernetes.io/mode":"EnsureExists"},"name":"standard"},"provisioner":"k8s.io/minikube-hostpath"}
,storageclass.kubernetes.io/is-default-class=true
Provisioner:           k8s.io/minikube-hostpath
Parameters:            <none>
AllowVolumeExpansion:  <unset>
MountOptions:          <none>
ReclaimPolicy:         Delete
VolumeBindingMode:     Immediate
Events:                <none>
1 Like
4

Could it be a permission issue on the host path? Also, if enabled, it’s good to check audit log for SELinux errors.

1 Like
5

Maybe, but minikube is creating these nodes (kvm2) so I’d expect it to work, really. How do people normally run this locally? I’ll look at host perms and minikube and see if I can find anything, though.

2 Likes
6

Have you found something? I get the same error.

1 Like
8

@jhackett are you running Kubernetes with Ceph?

@Ege_Gunes is there an option to run another init container to change the filesystem user?

1 Like
9

No, no Ceph. It is the hostpath storage driver. I’m going to try and compare with OpenEBS and see if that yields a different result from pointing at an iSCSI volume with its own filesystem. Same principle as using an RBD, I suppose.

I’m reasonably sure there is going to be something peculiar to the hostpath driver, but it isn’t clear what that will be. I haven’t had time to work on this as I’m on holiday this week but I’ll be trying the following:

  • Check an OpenEBS volume’s behaviour
  • RTFM about hostpath
  • Examine the Dockerfile for the relevant container and see in more detail what it is doing

I haven’t looked at this since last week, but I’m reasonably sure the failing step copies an entrypoint script into a shared directory. If so, I’d be interested to hear why it isn’t baked into the image instead. Could someone from the percona side explain the reasoning for this?

1 Like
10

OpenEBS volumes yield the same result. This left me a little bit puzzled, but having also tried on Scaleway’s Kubernetes Kapsule I’ve had some success. It may well be user error but it is unclear to me how.

Can anyone on the percona side point to some docs about how percona devs work on this? It’d be interesting to hear if there are some

1 Like
11

A little frustratingly, these containers work fine with emptydir: {} rather than the PVC claim when the standard storage class is set to be hostPath. It seems odd, but it is workable in both dev-like and prod-like storage setups now so my needs are met well enough.

I’m going to mark this as a solution and put something in my helm chart to feature flag emptydir storage for when I’m in dev.

1 Like
12

I’ve tried adding the emptydir, but seems it still uses PCVs

13

@jhackett I used openebs and tools like ondat in the past quite succesfully. They were all using hostpath provisioners. Can you share more details of what is not working for OpenEBS (versions, storageclass configs, underlying storage, etc)