Exposing MongoDB via Load balancer - switch from internet-facing to internal facing (AWS)

MongoDB Percona Operator for MongoDB
1

We have a Kubernetes cluster provisioned on AWS EKS. We have had a mongodb replicaset running for some time now using the percona mongodb operator and mongodb server (thank you).

We exposed our mongodb instance to be accessible via a load balancer , this load balancer scheme is set to internet facing by default.

    expose:
      enabled: true
      exposeType: LoadBalancer
      serviceAnnotations:
        service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing

Goal:

Due to security considerations we now need to switch this load balancer sceme from being internet-facing to be internal facing. Annotations - AWS Load Balancer Controller

I’m looking for some guidance on how we might go ahead and make this change.

Attempt:

i have setup a test environment and and changed the config to

service.beta.kubernetes.io/aws-load-balancer-scheme: internal

We then upgraded and a new revision was deployed.

helm upgrade psmdb-db percona/psmdb-db --namespace mongodb -f psmdb-db.values.yaml

However upon observing the load balancer we can see that it still remains as internet facing

Any help is appreciated.

image
image1061×417 19.7 KB

Update

I just checked the operator logs and i came across

2023-09-20T13:46:33.484Z ERROR Reconciler error {"controller": "psmdb-controller", "object": {"name":"psmdb-db","namespace":"mongodb"}, "namespace": "mongodb", "name": "psmdb-db", "reconcileID": "4c899e9e-bb3f-43b0-a060-f2ef7fec3104", "error": "failed to ensure services of replset rs0: failed to create external service for replset rs0: Service \"psmdb-db-rs0-0\" is invalid: spec.loadBalancerClass: Invalid value: \"null\": may not change once set", "errorVerbose": "failed to ensure services of replset rs0: failed to create external service for replset rs0: Service \"psmdb-db-rs0-0\" is invalid: spec.loadBalancerClass: Invalid value: \"null\": may not change once set\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:477\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1594"}
2

@Kay_Khan
Do you have mongos or it is just a replica set?

3

Just replicaset. I later learned that you cannot do this change in place and it will require recreating mongodb instance.

2023-09-20T13:46:33.484Z ERROR Reconciler error {"controller": "psmdb-controller", "object": {"name":"psmdb-db","namespace":"mongodb"}, "namespace": "mongodb", "name": "psmdb-db", "reconcileID": "4c899e9e-bb3f-43b0-a060-f2ef7fec3104", "error": "failed to ensure services of replset rs0: failed to create external service for replset rs0: Service \"psmdb-db-rs0-0\" is invalid: spec.loadBalancerClass: Invalid value: \"null\": may not change once set", "errorVerbose": "failed to ensure services of replset rs0: failed to create external service for replset rs0: Service \"psmdb-db-rs0-0\" is invalid: spec.loadBalancerClass: Invalid value: \"null\": may not change once set\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:477\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1594"}
4

Hi @Sergey_Pronin and @Kay_Khan,

I hope you have been well!

Do you have any example of how to set mongodb shard to expose mongos using internal loadbalancer?

I have already tried using Exposing the cluster - Percona Operator for MongoDB, but it seems to be not working to expose the DNS outside the cluster. Internally I am able to access mongos from other pods in the same cluster.

The custom-values.yaml is:

fullnameOverride: "mongodb"
sharding:
  mongos:
    size: 3
    expose:
      exposeType: ClusterIP
backup:
  enabled: false
5

Hello @Tiago_Barreto,

  1. Set expose type to “LoadBalancer”
  2. Set the annotation to use internal load balancer. Annotation depends on the vendor you use. Below is the example for AWS EKS:
sharding:
  mongos:
    size: 3
    expose:
      exposeType: LoadBalancer
      serviceAnnotations:
        service.beta.kubernetes.io/aws-load-balancer-type: "internal"
6

Thank you @Sergey_Pronin!