# Affinity and Mongos and CursorNotFound - oh my!
**URL:** https://forums.percona.com/t/affinity-and-mongos-and-cursornotfound-oh-my/11837
**Category:** Percona Operator for MongoDB
**Created:** [August 18, 2021, 6:57pm UTC](https://forums.percona.com/t/affinity-and-mongos-and-cursornotfound-oh-my/11837 "2021-08-18T18:57:09Z")
**Posts on this page:** 18
**Page:** 1
### Author:  [@Nick\_Cooper](https://forums.percona.com/u/Nick_Cooper)
#### Post date: [August 18, 2021, 6:57pm UTC](https://forums.percona.com/t/affinity-and-mongos-and-cursornotfound-oh-my/11837/1 "2021-08-18T18:57:09Z")
We have a fairly minimal setup with 3-cfg, 3-mongos, 3-mongod [all for rs0]; likely in future to grow. At our initial size though we are continually running into issues with CursorNotFound when scanning reasonably small collections (i.e. the timeouts and MB limits are not coming into play).
It appears our main issue is likely connection affinity, namely a given service uses the single ClusterIP for the mongos - but this is transparently round-robin’d between the mongos instances, and if that occurs mid-scan it results in CursorNotFound.
I’m wondering what others have done to resolve this?
_Potential solutions and my thoughts thus-far:_
1. Reduce Mongos to 1 - makes affinity irrelevant. This is workable today but I worry about scaling up later means i’m just pushing the problem to later.
2. Enable sessionAffinity: ClientIP for a mongo service. We actually tried this, it does appear to reduce the incidence of issue but does not completely eliminate it. This is because our kube-proxy is configured in iptables mode and thus there is a timeout of 3h (10800 sec) where it’ll switch as before.
3. Adjust kube-proxy to something with source-hash [e.g. ipvs-sh]; this would work but I see it as an extreme option as it affects all services, not solely mongos. Because of this I prefer not to do it.
_Alternative:_ Move to a more advanced network fabric that permits this routing configuration per-service. This is also an extreme option in my mind.
4. Have a service _per_ mongos (or semi-equivalently make it a StatefulSet not Deployment); this would have the effect of moving mongos instance selection into our client, which will not switch mid-operation and thus eliminate the problem.
My preference to solve this is #4 and this feels like the ‘correct’ solution generally - but it requires modification of the operator (or us moving away from it).
Wondering how other people have dealt with this?
---
### Author:  [@Sergey\_Pronin](https://forums.percona.com/u/Sergey_Pronin)
#### Post date: [August 23, 2021, 8:22am UTC](https://forums.percona.com/t/affinity-and-mongos-and-cursornotfound-oh-my/11837/2 "2021-08-23T08:22:20Z")
Hello @Nick_Cooper ,
thank you for submitting this.
Seems it is similar issue as described here: [[K8SPSMDB-347] support session affinity for mongos service - Percona JIRA](https://jira.percona.com/browse/K8SPSMDB-347)
But we have concluded, that it is an extremely rare case as the connection stays within single TCP session and should not be jumping between the nodes.
Are there any frequent mongos restarts in your cluster?
Do you have a good way to reproduce this issue?
---
### Author:  [@Nick\_Cooper](https://forums.percona.com/u/Nick_Cooper)
#### Post date: [August 23, 2021, 4:18pm UTC](https://forums.percona.com/t/affinity-and-mongos-and-cursornotfound-oh-my/11837/3 "2021-08-23T16:18:34Z")
Hello,
I do agree it is similar to that issue (likely the same), however I do not think that conclusion of being within a single TCP session is accurate. If kube-proxy is running in iptables mode the packet filter that results in affinity will be redirected about every 3hrs by default - i.e. the connection can freely jump between nodes if the client node is sufficiently long lived.
We do not have frequent mongos restarts [in fact they have been running approximately a month], however we notice increased likelihood of this error when our client-server is itself also long-lived - we do not have reliable reproduction steps beyond have continuous traffic from a long-lived client. We have also validated that pinning a connection to a single mongos does not exhibit the issue.
Our current work-around we’re looking to do is have a k8-aware client side library to expose all the mongos pods to the pymongo client (instead of the sole synthetic service address). This could be more neatly achieved if they had stable names/individual services which is likely our next step.
---
### Author:  [@Nick\_Cooper](https://forums.percona.com/u/Nick_Cooper)
#### Post date: [September 7, 2021, 4:56pm UTC](https://forums.percona.com/t/affinity-and-mongos-and-cursornotfound-oh-my/11837/4 "2021-09-07T16:56:22Z")
@Sergey_Pronin just wondering if you had any further thoughts on this?
If we had an experimental option to create N services we could test it in our cluster to validate if it solves the issue?
---
### Author:  [@Sergey\_Pronin](https://forums.percona.com/u/Sergey_Pronin)
#### Post date: [September 9, 2021, 3:50pm UTC](https://forums.percona.com/t/affinity-and-mongos-and-cursornotfound-oh-my/11837/5 "2021-09-09T15:50:24Z")
Hello @Nick_Cooper ,
sorry, dropped a ball here. Let me discuss it with our MongoDB team internally.
---
### Author:  [@Nick\_Cooper](https://forums.percona.com/u/Nick_Cooper)
#### Post date: [September 9, 2021, 4:04pm UTC](https://forums.percona.com/t/affinity-and-mongos-and-cursornotfound-oh-my/11837/6 "2021-09-09T16:04:41Z")
Thank you!
We decided to fork the operator and move to statefulsets. So will reply here with our findings if this fixes the problem (it is a rare error so will take some time to validate)
---
### Author:  [@Nick\_Cooper](https://forums.percona.com/u/Nick_Cooper)
#### Post date: [September 17, 2021, 5:10pm UTC](https://forums.percona.com/t/affinity-and-mongos-and-cursornotfound-oh-my/11837/7 "2021-09-17T17:10:11Z")
As a heads up, since we migrated to our forked operator - we have seen zero incidents of this.
Our change was to make the mongos a statefulset [not deployment]; and then in pymongo give it a direct reference to each member. As noted above I believe this is an issue for any long-lived client that may see unexpected switching after the 3hr mark on k8
---
### Author:  [@Sergey\_Pronin](https://forums.percona.com/u/Sergey_Pronin)
#### Post date: [September 20, 2021, 8:03am UTC](https://forums.percona.com/t/affinity-and-mongos-and-cursornotfound-oh-my/11837/8 "2021-09-20T08:03:40Z")
Hello @Nick_Cooper .
Thank you for sharing. This means that you need a service per mongos pod, right?
---
### Author:  [@Nick\_Cooper](https://forums.percona.com/u/Nick_Cooper)
#### Post date: [September 20, 2021, 3:31pm UTC](https://forums.percona.com/t/affinity-and-mongos-and-cursornotfound-oh-my/11837/9 "2021-09-20T15:31:26Z")
Actually no, having just the single service (as with the current operator) is fine; as statefulsets gain then stable names, we have:
```auto
MONGO_HOST = [
"mongo-mongos-0.mongo-mongos",
"mongo-mongos-1.mongo-mongos",
"mongo-mongos-2.mongo-mongos",
]
```
(Our mongo instance is named ‘mongo’)
---
### Author:  [@Nick\_Cooper](https://forums.percona.com/u/Nick_Cooper)
#### Post date: [October 11, 2021, 4:12pm UTC](https://forums.percona.com/t/affinity-and-mongos-and-cursornotfound-oh-my/11837/10 "2021-10-11T16:12:31Z")
Hello,
Just figured i’d check in again and mention that we have not seen this error at all since Sept 9th [when we forked the operator and had the client be aware of all three].
I’m now quite confident that the transparent proxy mode of K8 is at-fault here, and all clients should use stable names to _all_ mongos instances,
Thanks,
Nick.
---
### Author:  [@Sergey\_Pronin](https://forums.percona.com/u/Sergey_Pronin)
#### Post date: [October 12, 2021, 1:31pm UTC](https://forums.percona.com/t/affinity-and-mongos-and-cursornotfound-oh-my/11837/11 "2021-10-12T13:31:34Z")
Hello @Nick_Cooper - thank you for sharing.
What do you think about using [Service SessionAffinity](https://kubernetes.io/docs/concepts/services-networking/service/) flag for this?
---
### Author:  [@sohahm](https://forums.percona.com/u/sohahm)
#### Post date: [October 12, 2022, 4:50pm UTC](https://forums.percona.com/t/affinity-and-mongos-and-cursornotfound-oh-my/11837/12 "2022-10-12T16:50:58Z")
@Nick_Cooper @Sergey_Pronin Please advise how can I set **sessionAffinity: ClientIP**. I do not see it in the chart values.yaml. I am having the same issue of CursorNotFound.
---
### Author:  [@Sergey\_Pronin](https://forums.percona.com/u/Sergey_Pronin)
#### Post date: [October 13, 2022, 12:04pm UTC](https://forums.percona.com/t/affinity-and-mongos-and-cursornotfound-oh-my/11837/13 "2022-10-13T12:04:55Z")
Hello @sohahm ,
since version 1.12.0 of the Operator we run mongos as a statefulset and allow users to expose them through a service per pod. In that case your database client will take care of Cursor tracking.
See this option for more details: [Custom Resource options - Percona Operator for MongoDB](https://docs.percona.com/percona-operator-for-mongodb/operator.html#sharding-mongos-expose-serviceperpod)
And this JIRA ticket is the one delivering it: [[K8SPSMDB-599] Multi-thread transaction failure when using the default mongos ClusterIP service - Percona JIRA](https://jira.percona.com/browse/K8SPSMDB-599)
Please let me know if you still have questions.
---
### Author:  [@Domenic\_Bove](https://forums.percona.com/u/Domenic_Bove)
#### Post date: [December 11, 2023, 7:32pm UTC](https://forums.percona.com/t/affinity-and-mongos-and-cursornotfound-oh-my/11837/14 "2023-12-11T19:32:08Z")
Hey thanks for this addition!
What do you thinking about updating the [Exposing the Cluster Docs](https://docs.percona.com/percona-operator-for-mongodb/expose.html).
I installed with default settings (ie servicePerPod set to false) and used the documented connection string `mongodb://userAdmin:userAdminPassword@my-cluster-name-mongos..svc.cluster.local/admin?ssl=false`
And I thought the cluster was healthy, eventually running into the CurserNotFound error in a client application.
After updating `sharding.mongos.expose.servicePerPod: true`, I then had to update my connection string to `mongodb://userAdmin:userAdminPassword@mongodb-mongos-0..svc.cluster.local:27017,mongodb-mongos-1..svc.cluster.local:27017/admin?ssl=false`
Would be nice if the docs warned me about the error!
---
### Author:  [@Sergey\_Pronin](https://forums.percona.com/u/Sergey_Pronin)
#### Post date: [December 14, 2023, 9:14am UTC](https://forums.percona.com/t/affinity-and-mongos-and-cursornotfound-oh-my/11837/15 "2023-12-14T09:14:05Z")
@Domenic_Bove - yes, I think it makes sense. Thank you for the suggestion.
We will document it here: [[K8SPSMDB-1026] Document Service per Pod for mongos - Percona JIRA](https://jira.percona.com/browse/K8SPSMDB-1026)
---
### Author:  [@Thang\_Mai](https://forums.percona.com/u/Thang_Mai)
#### Post date: [October 22, 2024, 7:32am UTC](https://forums.percona.com/t/affinity-and-mongos-and-cursornotfound-oh-my/11837/16 "2024-10-22T07:32:14Z")
Hello, I am currently facing the same issue. I have two pods for `mongos`, and our system uses Lambda, which is in the same VPC as the Mongo cluster but cannot access it directly. Therefore, I created a service (NodePort) for `mongos`. I also use CloudMap + External DNS to create DNS records for the Node IPs.
When I use `sharding.mongos.expose.servicePerPod: true`, there will be two services with the same annotations, which causes an error for External DNS when two services expose the same URL. Is there a way I can customize the annotations for each service?
---
### Author:  [@Sergey\_Pronin](https://forums.percona.com/u/Sergey_Pronin)
#### Post date: [October 25, 2024, 12:44pm UTC](https://forums.percona.com/t/affinity-and-mongos-and-cursornotfound-oh-my/11837/17 "2024-10-25T12:44:59Z")
@Thang_Mai how would you customize it? Do you want to have a separate domain name per Mongos Pod/Service?
I’m curious how does it work for other products that have multiple Service resources.
Also, is it imperative for you to use `ServicePerPod`?
---
### Author:  [@Thang\_Mai](https://forums.percona.com/u/Thang_Mai)
#### Post date: [October 28, 2024, 2:42am UTC](https://forums.percona.com/t/affinity-and-mongos-and-cursornotfound-oh-my/11837/18 "2024-10-28T02:42:03Z")
@Sergey_Pronin Yes, I think we should have a separate domain name per Mongos Pod/Service.
I am currently getting `CurserNotFound` errors so I think using service per pod will be the solution for us. But applying the above solution is causing problems with our external dns and cloud map, multiple services using the same domain will cause problems for external dns when they try to override each other.