Percona mongoDB crash on AKS

MongoDB Percona Operator for MongoDB
1

Hello,

I’m looking for installing Percona MongoDB on my AKS Cluster. I read the documentation, i installed the operator and download the cr.yaml.

Before apply it, i modify the sharded mode to false and deployed it.

The deployement is succesfull. i get the ready status when i type “kubectl get psmdb”

but after 2-3 minutes i got this error on each pods :

Liveness probe failed: command “/opt/percona/mongodb-healthcheck k8s liveness --ssl --sslInsecure --sslCAFile /etc/mongodb-ssl/ca.crt --sslPEMKeyFile /tmp/tls.pem --startupDelaySeconds 7200” timed out

so i can’t work with percona mongodb and i didn’t found any solutions.

Could you help me :slight_smile:

Thanks a lot

Maxence

2

Hello @MaxSai ,

under normal circumstance I can’t reproduce it. Is there anything specific about your cluster or deployment?

Like, it might be that your compute nodes are too small and they don’t have enough resources, so mongod can’t process requests.

3

Hello @Sergey_Pronin,

I’ve got a cluster with 4 nodes 2vcpu/8go ram.

For the deployement i juste deactivate sharding

i’m in a private network with firewall, and i use “private aks option”. is it possible that access to the network is blocked with one of your endpoint for deployement ?

4

Hello,

i’ve done others test.

i follow the tutorial with public aks and percona mongodb start succesfully (sharded mode).

so i tried to make the same aks with private access, we checked with network team for network issues in firewall. and nothing showed up.

Do you know if percona may work in closed environnement ?

Best regards.

Maxence

5

Yes, it can. There are no limitations to that from Operator or DB itself. It can also work in air-gapped environment.

When you say “private” - do you mean this: Create a private Azure Kubernetes Service (AKS) cluster - Azure Kubernetes Service | Microsoft Learn ?

Without having enough details about your networking and traffic restirictions, it would be just guessing. If you have any sort of network policies or mesh/proxying - I would start from there.

6

Hello Sergey,

Yes by private i mean this, the node of my aks are on private ip. we have an hub & spoke infrastucture with vnet peering. we have azure route table with rules to redirect some traffic to our firewall.

Whole internet network (0.0.0.0/0) redirect to our firewall
all interactions outside my virtual network is also redirect to firewall

by default we close whole traffic, we oppened rules to get docker images. and check.percona.com.

As Sharded or replicaset, my database is deployed but still crash on liveness probes on mongod pods.

So the cluster don’t stay in ready state.

Do you have any Idea ?

Other question, we seen ur beta “Percona Everest”, this project may run on azure kubernetes services even if its not in cloud provider recommanded ?

Best regards Maxence

.

7

Hello @MaxSai ,

the problem is that Operator can’t connect to cluster. This might be due to some networking restrictions.
Are there any network policies in your cluster? Or traffic firewalls that restrict traffic flows between the pods?

As for Percona Everest - it can run on any Kubernetes cluster. Right now we rigorously test it on AWS EKS and GKE, but as our Operators are certified on AKS, it will work too.

@Peter_Szczepaniak is a product manager for Everest and would be glad to give you a demo or tell more about it :slight_smile:

8

Hi, I tried installing Percona Everest on AKS and it worked.
But officially we don’t support it yet.

9

Hello Serguey,

i’ve got good news, percona is stable on my aks.

for details, we have 2 aks on one vnet, when we use private option, for routing aks use azure route table and write pod cicdr in this route table, but all my aks are on same vnet with same pod cicdr so they try to have priority on route table, so pod don’t know how to make liveness probes.

I change pod cicdr on second aks and it work :tada: Thanks a lot for your help !

@Peter_Szczepaniak i think we gonna contact u for a demo. but i have few questions before

  • Do you have a public roadmap ? Do you know when u i’ll support azure storage for backup ? and when we i’ll be able to use private load balancer with vnet IP ?

Best regards !

Maxence

1 Like
10

Hey Max,

I am sure happy to hear that you’ve been successful with Everest. Please send your availability to my email peter.szczepaniak@percona.com and I’ll set up zoom where we can discuss your needs and feedback. We do not currently have public roadmap, we recently published our JIRA project and hopefully soon we will be able to make our roadmap available as well.

Best!
Peter

1 Like