There have been different approaches to host a lot of databases in a single MongoDB server. As many tickets in MongoDB Jira already show, there are still issues.
In our project the goal is to host at least 10M databases and now it looks like we have found a solution:
Lets say you name the databases with numbers from 1-n. And naively one would create x=3 MongoDB clusters. Then these databases can be distributed to the clusters by using n mod x. Obviously if n is getting high, there will be too many databases/collection per MongoDB cluster.
We have found that 50’000 dbs each with approx 10 coll (=1M coll+index files) can be hosted on a single cluster. A restart (after a crash or pod rescheduling) would take approx 15min.
So in order to host 10M db we would need 200 clusters - which would make Percona happy, if you have a support contract. Nevertheless it would be feasible to manage them - but it takes a lot of resources.
To avoid that, we are swapping out the collections (=backup) if they are not needed and since our collections are small, this can be done on the fly (restore) and with a CronJob that runs every couple of minutes for backup.
So it looks like with 16-32 clusters you are able to host 10M dbs and this system is incredible fast (especially for writes) - because of the additional parallelization.
NAME READY STATUS RESTARTS AGE
pod/mongodb00-cluster-rs-0 1/1 Running 0 4m33s
pod/mongodb00-cluster-rs-1 1/1 Running 0 3m51s
pod/mongodb00-cluster-rs-2 1/1 Running 0 3m12s
pod/mongodb01-cluster-rs-0 1/1 Running 0 4m29s
pod/mongodb01-cluster-rs-1 1/1 Running 0 3m44s
pod/mongodb01-cluster-rs-2 1/1 Running 0 2m59s
pod/mongodb02-cluster-rs-0 1/1 Running 0 4m25s
pod/mongodb02-cluster-rs-1 1/1 Running 0 3m40s
pod/mongodb02-cluster-rs-2 1/1 Running 0 3m1s
pod/mongodb03-cluster-rs-0 1/1 Running 0 4m22s
pod/mongodb03-cluster-rs-1 1/1 Running 0 3m34s
pod/mongodb03-cluster-rs-2 1/1 Running 0 2m52s
pod/mongodb04-cluster-rs-0 1/1 Running 0 4m19s
pod/mongodb04-cluster-rs-1 1/1 Running 0 3m34s
pod/mongodb04-cluster-rs-2 1/1 Running 0 2m48s
pod/mongodb05-cluster-rs-0 1/1 Running 0 4m14s
pod/mongodb05-cluster-rs-1 1/1 Running 0 3m26s
pod/mongodb05-cluster-rs-2 1/1 Running 0 2m35s
pod/mongodb06-cluster-rs-0 1/1 Running 0 4m11s
pod/mongodb06-cluster-rs-1 1/1 Running 0 3m22s
pod/mongodb06-cluster-rs-2 1/1 Running 0 2m37s
pod/mongodb07-cluster-rs-0 1/1 Running 0 4m8s
pod/mongodb07-cluster-rs-1 1/1 Running 0 3m20s
pod/mongodb07-cluster-rs-2 1/1 Running 0 2m28s
...
pod/percona-server-mongodb-operator-f674f67b7-q9w6g 1/1 Running 0 91m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/mongodb00-cluster-rs ClusterIP None <none> 27017/TCP 4m34s
service/mongodb01-cluster-rs ClusterIP None <none> 27017/TCP 4m30s
service/mongodb02-cluster-rs ClusterIP None <none> 27017/TCP 4m26s
service/mongodb03-cluster-rs ClusterIP None <none> 27017/TCP 4m23s
service/mongodb04-cluster-rs ClusterIP None <none> 27017/TCP 4m20s
service/mongodb05-cluster-rs ClusterIP None <none> 27017/TCP 4m15s
service/mongodb06-cluster-rs ClusterIP None <none> 27017/TCP 4m12s
service/mongodb07-cluster-rs ClusterIP None <none> 27017/TCP 4m9s
...
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/percona-server-mongodb-operator 1/1 1 1 91m
NAME DESIRED CURRENT READY AGE
replicaset.apps/percona-server-mongodb-operator-f674f67b7 1 1 1 91m
NAME READY AGE
statefulset.apps/mongodb00-cluster-rs 3/3 4m34s
statefulset.apps/mongodb01-cluster-rs 3/3 4m30s
statefulset.apps/mongodb02-cluster-rs 3/3 4m26s
statefulset.apps/mongodb03-cluster-rs 3/3 4m23s
statefulset.apps/mongodb04-cluster-rs 3/3 4m20s
statefulset.apps/mongodb05-cluster-rs 3/3 4m15s
statefulset.apps/mongodb06-cluster-rs 3/3 4m12s
statefulset.apps/mongodb07-cluster-rs 3/3 4m9s
...
Happy New Year !