A lot of count of threads in mongo from operator deployment

MongoDB Percona Operator for MongoDB
, ,
1

Hi. I researched problem from this topic and found another problem - a lot of count of threads in mongod process, up to 30k. I researched this problem and found out if I drop operator deployment than count of threads decrease down to 150-200 threads. But I don’t know wny it is happen.
I take yaml-file for deploy percona-operator deployment from github and I don’t change it for my deploy.
And I deploy mongodb by cr.yaml and I change in this file settings for gettings replicaset - limits, requests and sharding = false.
Probably, I should change any settings for operator deployment, but I do not know which ones.

before_delete_operator
before_delete_operator799×455 74.9 KB

after_delete_operator
after_delete_operator838×451 76.3 KB

1 Like
2

Added output of pt-k8s-debug-collector and screenshots of threads and pods at the moment
pt-k8s-debug-collector_output.log (11.6 KB)

pods
pods949×135 19.4 KB

threads
threads864×449 81.3 KB

When I got output, my cluster was no load.

1 Like
3

Added archive with data from minikube cluster. I noticed same problem there.
Results
I will add result from cloud cluster later.

1 Like
4

Hey @nikita.savchenko , I cannot reproduce it.

  1. Take 1.10 branch from github
  2. Deploy bundle.yaml
  3. Deploy cr.yaml with disabled sharding
  4. If I login into any pod I do not see more than 90 threads and this number is not growing over time.

Is there anything else that I should do?
Is the cluster healthy or you see some container restarts (I’m implying some health checks failures)?

P.S. tried it for 1.11 (main branch) - cannot reproduce it as well

1 Like
5

Hi, @Sergey_Pronin ! Thanks for answer! It’s strange that you don’t have the same problem, but I do. Unfortunately, I’m busy now and can retry problem later.
Tell me, please, how did you check the number of threads?
I do the same sequence of actions and I see increasing of threads count by linux top inside containers mongod and by pmm-monitoring (increasing count of connections).

1 Like
6

@nikita.savchenko as I mentioned - just regular deployment with defaults. Nothing fancy.
Do you have pmm enabled in the CR? Anything else enabled to that I can try this out?

1 Like
7

Hi, @Sergey_Pronin!
Results of last retrying here
Pmm monitoring is not enabled, but I added output from db.serverStatus().connections on master-replica.

1 Like
8

I was able to reproduce it.
It happens when setFCV flag is set to true. Not sure why you set it to true and not sure why is it causing this yet.

  upgradeOptions:
    versionServiceEndpoint: https://check.percona.com
    apply: 4.4-recommended
    schedule: "0 2 * * *"
    setFCV: true

setFCV flag is used only for major version upgrade, so I’m not sure how it can be causing this.

Here is the ticket to get to the root of it: [K8SPSMDB-602] setFCV flag causes a thread leak - Percona JIRA

2 Likes
9

Probably this one is related to one more bug: K8SPSMDB-650: Reconcile should not be scheduled until operator has work to do by kvaster · Pull Request #880 · percona/percona-server-mongodb-operator · GitHub

1 Like