callum
November 28, 2024, 2:47am
1
I’m trying to setup percona-server-mongodb into a k8s cluster running rancher. I’ve successfully used a helmfile for this into regular k8s clusters with no issues but I’m struggling into a Rancher setup.
The Operator installs without issue, when helm installs percona-server-mongodb it outputs like everything worked fine, even with --debug. The only resources that are created however are the PerconaServerMongoDB resource. The service and pods are never created. The PerconaServerMongoDB also gets stuck in Removing right after helm installs it.
The operator reports no errors either. Any assistance would be greatly appreciated as I feel I’ve exhausted every log source I can think of.
Helmchart version: psmdb-db-1.15.3
My yaml file looks like so:
crVersion: 1.15.0
allowUnsafeConfigurations: false
image:
repository: percona/percona-server-mongodb
tag: 6.0.12-multi
imagePullPolicy: IfNotPresent
# imagePullSecrets: []
# tls:
# # 90 days in hours
# certValidityDuration: 2160h
secrets:
# If you set users secret here the operator will use existing one or generate random values
# If not set the operator generates the default secret with name <cluster_name>-secrets
users: li-config-mongodb-secrets
# encryptionKey: my-cluster-name-mongodb-encryption-key
pmm:
enabled: false
image:
repository: percona/pmm-client
tag: 2.41.0
serverHost: monitoring-service
replsets:
- name: rs0
size: 3
affinity:
antiAffinityTopologyKey: "kubernetes.io/hostname"
podDisruptionBudget:
maxUnavailable: 1
expose:
enabled: false
exposeType: ClusterIP
resources:
limits:
cpu: 2000m
memory: 8G
requests:
cpu: 300m
memory: 0.5G
volumeSpec:
pvc:
# storageClassName: standard
# accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: 32Gi
nonvoting:
enabled: false
size: 3
affinity:
antiAffinityTopologyKey: "kubernetes.io/hostname"
podDisruptionBudget:
maxUnavailable: 1
resources:
limits:
cpu: 300m
memory: 0.5G
requests:
cpu: 300m
memory: 0.5G
volumeSpec:
pvc:
# storageClassName: standard
# accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: 3Gi
arbiter:
enabled: false
size: 1
affinity:
antiAffinityTopologyKey: "kubernetes.io/hostname"
sharding:
enabled: false
balancer:
enabled: false
configrs:
size: 3
affinity:
antiAffinityTopologyKey: "kubernetes.io/hostname"
podDisruptionBudget:
maxUnavailable: 1
expose:
enabled: false
exposeType: ClusterIP
resources:
limits:
cpu: 300m
memory: 0.5G
requests:
cpu: 300m
memory: 0.5G
volumeSpec:
pvc:
# storageClassName: standard
# accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: 3Gi
mongos:
size: 2
affinity:
antiAffinityTopologyKey: "kubernetes.io/hostname"
podDisruptionBudget:
maxUnavailable: 1
resources:
limits:
cpu: 300m
memory: 0.5G
requests:
cpu: 300m
memory: 0.5G
expose:
exposeType: ClusterIP
backup:
enabled: false
image:
repository: percona/percona-backup-mongodb
tag: 2.3.0-multi
serviceAccountName: percona-server-mongodb-operator
Hey @callum - you need to check the following:
The status of psmdb object: kubectl get psmdb
If StatefulSets were created: kubectl get sts
If statefulsets are created, but Pods are not being created - it can be for various reasons and needs deeper investigation. Try describing one of the statefulsets: kubectl describe sts NAME
callum
November 28, 2024, 11:18pm
3
The status of psmdb is:
> kubectl get psmdb -n namespace
NAME ENDPOINT STATUS AGE
mongodb 44h
There are no StatefulSets being created at all currently.
@callum okay, so seems it is not processed or you have some permission issues.
kubectl describe psmdb -n namespace
is operator pod running?
kubectl get pods -n namespace
what is in the logs of the operator?
kubectl logs ...
Please give me at least something to help you.
callum
December 4, 2024, 4:32am
5
Here is the kubectl describe on psmdb
$ kubectl describe psmdb -n namespace
Name: mongodb
Namespace: namespace
Labels: app.kubernetes.io/instance=mongodb
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=psmdb-db
app.kubernetes.io/version=1.15.0
helm.sh/chart=psmdb-db-1.15.3
Annotations: meta.helm.sh/release-name: mongodb
meta.helm.sh/release-namespace: namespace
API Version: psmdb.percona.com/v1
Kind: PerconaServerMongoDB
Metadata:
Creation Timestamp: 2024-11-27T03:05:18Z
Deletion Grace Period Seconds: 0
Deletion Timestamp: 2024-11-27T04:41:11Z
Finalizers:
delete-psmdb-pods-in-order
Generate Name: mongodb
Generation: 2
Resource Version: 158887919
UID: 14c95d98-beae-4b9c-87c2-ca99fcd0bf52
Spec:
Backup:
Enabled: false
Image: percona/percona-backup-mongodb:2.3.0-multi
Pitr:
Enabled: false
Service Account Name: percona-server-mongodb-operator
Cr Version: 1.15.0
Image: percona/percona-server-mongodb:6.0.12
Image Pull Policy: IfNotPresent
Multi Cluster:
Enabled: false
Pause: false
Pmm:
Enabled: false
Image: percona/pmm-client:2.41.0
Server Host: monitoring-service
Replsets:
Affinity:
Anti Affinity Topology Key: kubernetes.io/hostname
Arbiter:
Affinity:
Anti Affinity Topology Key: kubernetes.io/hostname
Enabled: false
Size: 1
Expose:
Enabled: true
Expose Type: ClusterIP
Name: rs0
Nonvoting:
Affinity:
Anti Affinity Topology Key: kubernetes.io/hostname
Enabled: false
Pod Disruption Budget:
Max Unavailable: 1
Resources:
Limits:
Cpu: 300m
Memory: 0.5G
Requests:
Cpu: 300m
Memory: 0.5G
Size: 3
Volume Spec:
Persistent Volume Claim:
Resources:
Requests:
Storage: 3Gi
Pod Disruption Budget:
Max Unavailable: 1
Resources:
Limits:
Cpu: 2000m
Memory: 8G
Requests:
Cpu: 300m
Memory: 0.5G
Size: 3
Volume Spec:
Persistent Volume Claim:
Resources:
Requests:
Storage: 32Gi
Secrets:
Users: mongodb-secrets
Sharding:
Balancer:
Enabled: false
Configsvr Repl Set:
Affinity:
Anti Affinity Topology Key: kubernetes.io/hostname
Expose:
Enabled: false
Expose Type: ClusterIP
Pod Disruption Budget:
Max Unavailable: 1
Resources:
Limits:
Cpu: 300m
Memory: 0.5G
Requests:
Cpu: 300m
Memory: 0.5G
Size: 3
Volume Spec:
Persistent Volume Claim:
Resources:
Requests:
Storage: 3Gi
Enabled: false
Mongos:
Affinity:
Anti Affinity Topology Key: kubernetes.io/hostname
Expose:
Expose Type: ClusterIP
Pod Disruption Budget:
Max Unavailable: 1
Resources:
Limits:
Cpu: 300m
Memory: 0.5G
Requests:
Cpu: 300m
Memory: 0.5G
Size: 2
Unmanaged: false
Update Strategy: SmartUpdate
Upgrade Options:
Apply: disabled
Schedule: 0 2 * * *
Set FCV: false
Version Service Endpoint: https://check.percona.com
Events:
The operator is running in another namespace, called operators.
$ kubectl get pods -n operators
NAME READY STATUS RESTARTS AGE
psmdb-operator-684b798654-f2gml 1/1 Running 23 (8h ago) 5d20h
And this is its logs:
$ kubectl logs psmdb-operator-684b798654-f2gml -n operators
2024-12-03T20:01:43.236Z INFO setup Manager starting up {"gitCommit": "ed2d8b4907c39beadfb020ce1cb555fee0ac682d", "gitBranch": "release-1-15-0", "goVersion": "go1.20.9", "os": "linux", "arch": "amd64"}
2024-12-03T20:01:43.274Z INFO server version {"platform": "kubernetes", "version": "v1.30.6+rke2r1"}
2024-12-03T20:01:43.304Z INFO controller-runtime.metrics Starting metrics server
2024-12-03T20:01:43.305Z INFO starting server {"kind": "health probe", "addr": "[::]:8081"}
2024-12-03T20:01:43.305Z INFO controller-runtime.metrics Serving metrics server {"bindAddress": ":8080", "secure": false}
I1203 20:01:43.307311 1 leaderelection.go:250] attempting to acquire leader lease operators/08db0feb.percona.com...
I1203 20:01:59.446760 1 leaderelection.go:260] successfully acquired lease operators/08db0feb.percona.com
2024-12-03T20:01:59.448Z INFO Starting EventSource {"controller": "psmdb-controller", "source": "kind source: *v1.PerconaServerMongoDB"}
2024-12-03T20:01:59.448Z INFO Starting Controller {"controller": "psmdb-controller"}
2024-12-03T20:01:59.449Z INFO Starting EventSource {"controller": "psmdbrestore-controller", "source": "kind source: *v1.PerconaServerMongoDBRestore"}
2024-12-03T20:01:59.449Z INFO Starting EventSource {"controller": "psmdbrestore-controller", "source": "kind source: *v1.Pod"}
2024-12-03T20:01:59.449Z INFO Starting Controller {"controller": "psmdbrestore-controller"}
2024-12-03T20:01:59.449Z INFO Starting EventSource {"controller": "psmdbbackup-controller", "source": "kind source: *v1.PerconaServerMongoDBBackup"}
2024-12-03T20:01:59.449Z INFO Starting EventSource {"controller": "psmdbbackup-controller", "source": "kind source: *v1.Pod"}
2024-12-03T20:01:59.449Z INFO Starting Controller {"controller": "psmdbbackup-controller"}
2024-12-03T20:01:59.649Z INFO Starting workers {"controller": "psmdbbackup-controller", "worker count": 1}
2024-12-03T20:01:59.664Z INFO Starting workers {"controller": "psmdb-controller", "worker count": 1}
2024-12-03T20:01:59.670Z INFO Starting workers {"controller": "psmdbrestore-controller", "worker count": 1}
@callum by default our operators are deployed in a namespace scope. That way they can only manage the clusters and custom resources that were created in the same namespace where operator pod is deployed. We do it for security reasons, as that way you don’t need cluster roles.
You can easily deploy the operator in the cluster-wide mode. Just read this doc: Install Percona Server for MongoDB in multi-namespace (cluster-wide) mode - Percona Operator for MongoDB