splitHorizons not working with Percona Operator for MongoDB 1.15.0

Louis_Laborde · October 12, 2023, 11:59pm

Description:

MongoDB deployed with Percona Operator with split-horizon DNS doesn’t show horizons in rs.conf().

Steps to Reproduce:

Installed with Helm Percona Operator for MongoDB 1.15.0 on 3-node K3s cluster.

Installed with Helm MongoDB server 6.0.9-7 with with values.yaml file:
replsets:

name: rs
size: 3
sharding:
enabled: false
backup:
enabled: false

Updated with Helm MongoDB server 6.0.9-7 to enable split-horizon DNS with values.yaml file:
replsets:

name: rs
size: 3
splitHorizons:
db-psmdb-db-rs-0:
external: <node1.external.domain>
db-psmdb-db-rs-1:
external: <node2.external.domain>
db-psmdb-db-rs-2:
external: <node3.external.domain>
expose:
enabled: true
exposeType: ClusterIP
sharding:
enabled: false
backup:
enabled: false

Used mongosh to log into admin DB and dumped RS config:
rs [primary] admin> rs.conf()
{
_id: ‘rs’,
version: 8,
term: 1,
members: [
{
_id: 0,
host: ‘db-psmdb-db-rs-0.db-psmdb-db-rs.percona.svc.cluster.local:27017’,
arbiterOnly: false,
buildIndexes: true,
hidden: false,
priority: 2,
tags: { podName: ‘db-psmdb-db-rs-0’, serviceName: ‘db-psmdb-db’ },
secondaryDelaySecs: Long(“0”),
votes: 1
},
{
_id: 1,
host: ‘db-psmdb-db-rs-1.db-psmdb-db-rs.percona.svc.cluster.local:27017’,
arbiterOnly: false,
buildIndexes: true,
hidden: false,
priority: 2,
tags: { podName: ‘db-psmdb-db-rs-1’, serviceName: ‘db-psmdb-db’ },
secondaryDelaySecs: Long(“0”),
votes: 1
},
{
_id: 2,
host: ‘db-psmdb-db-rs-2.db-psmdb-db-rs.percona.svc.cluster.local:27017’,
arbiterOnly: false,
buildIndexes: true,
hidden: false,
priority: 2,
tags: { podName: ‘db-psmdb-db-rs-2’, serviceName: ‘db-psmdb-db’ },
secondaryDelaySecs: Long(“0”),
votes: 1
}
],
protocolVersion: Long(“1”),
writeConcernMajorityJournalDefault: true,
settings: {
chainingAllowed: true,
heartbeatIntervalMillis: 2000,
heartbeatTimeoutSecs: 10,
electionTimeoutMillis: 10000,
catchUpTimeoutMillis: -1,
catchUpTakeoverDelayMillis: 30000,
getLastErrorModes: {},
getLastErrorDefaults: { w: 1, wtimeout: 0 },
replicaSetId: ObjectId(“65282d6d43cc5ca40a5ee9f6”)
}
}

Each member should have displayed an “horizons” field.

Version:

MongoDB Operator 1.15.0 and MongoSB Server 6.0.9-7

Logs:

[If applicable, include any relevant log files or error messages]

Expected Result:

Each RS member should have an “horizons” field configured.

Actual Result:

Missing “horizons” field.

Additional Information:

[Include any additional information that could be helpful to diagnose the issue, such as browser or device information]

Tomislav_Plavcic · October 13, 2023, 12:39pm

Hi @Louis_Laborde !

I think you hit a bug reported here: [K8SPSMDB-994] cannot start a cluster with splitHorizon enabled from the begining - Percona JIRA
You can check your operator logs to see if there is an error for splitHorizon.
So you will probably need to start a cluster without splitHorizon and then update it.
This should be fixed in the next release.

Also I would advise you to check other limitations listed here at the end: Exposing the cluster - Percona Operator for MongoDB

Louis_Laborde · October 13, 2023, 2:57pm

Hello @Tomislav_Plavcic, maybe my description wasn’t clear, but I had done what you suggested, i.e. first helm-install WITHOUT spliHorizons configured, and then helm-upgrade WITH splitHorizons configured.

Tomislav_Plavcic · October 13, 2023, 4:03pm

Hi @Louis_Laborde !
Your description was fine, I need to read more carefully.
So it seems it works for me and let’s try to find the difference in your case.

I installed helm chart like this (exposed, but without splitHorizon):
helm install db-psmdb-db --namespace test ./psmdb-db --values psmdb-db/values.yaml
then after cluster got ready I added splitHorizon in chart values like this:

    splitHorizons:
      db-psmdb-db-rs0-0:
        external: rs0-0.mycluster.xyz
      db-psmdb-db-rs0-1:
        external: rs0-1.mycluster.xyz
      db-psmdb-db-rs0-2:
        external: rs0-2.mycluster.xyz
    expose:
      enabled: true
      exposeType: ClusterIP

and upgraded like this:
helm upgrade db-psmdb-db --namespace test ./psmdb-db --values psmdb-db/values.yaml

after that I see this line in operator logs:
2023-10-13T15:44:24.559Z INFO Updating horizons {"controller": "psmdb-controller", "object": {"name":"db-psmdb-db","namespace":"test"}, "namespace": "test", "name": "db-psmdb-db", "reconcileID": "1f11b46b-e305-44a3-834c-99c88f327070", "replset": "rs0"}

and in mongodb rs config I see this:

	"members" : [
		{
			"_id" : 0,
			"host" : "db-psmdb-db-rs0-0.db-psmdb-db-rs0.test.svc.cluster.local:27017",
			"arbiterOnly" : false,
			"buildIndexes" : true,
			"hidden" : false,
			"priority" : 2,
			"tags" : {
				"podName" : "db-psmdb-db-rs0-0",
				"serviceName" : "db-psmdb-db"
			},
			"horizons" : {
				"external" : "rs0-0.mycluster.xyz:27017"
			},
			"secondaryDelaySecs" : NumberLong(0),
			"votes" : 1
		},

So currently what I’m suspecting is that your helm upgrade didn’t go well and splitHorizons was not even added to custom resource.
If splitHorizons was added to custom resource your should see the line in operator logs like the one above (Updating horizons…) or there should be some error in operator logs.
You can check helm upgrade command with --dry-run option to see what yaml it will generate.

Also it would be helpful if you could add your helm install and upgrade commands to the output (I installed from helm chart repository locally from git, but if you installed from remote helm repo it might work differently).

Louis_Laborde · October 13, 2023, 10:12pm

Hello @Tomislav_Plavcic, I followed your steps exactly and still didn’t see the “horizons” fields in rs.conf().

Another “interesting” observation is that if I try to modify my PerconaServerMongoDB resource with kubectl instead of helm, passing in the modified official manifest “https://raw.githubusercontent.com/percona/percona-server-mongodb-operator/v1.15.0/deploy/cr.yaml” to enable “splitHorizons”, “kubectl -n -f cr.yaml” fails with output shown below.

strict decoding error: unknown field “spec.replsets[0].splitHorizons”

Some decoder “somewhe” doesn’t seem to like “splitHorizons” field name…

[root@adevxp035-c1n1 percona]# kubectl -n percona apply -f percona-db-manifs.1.15.0-mod2.yaml
The request is invalid: patch: Invalid value: “{"apiVersion":"psmdb.percona.com/v1","kind":"PerconaServerMongoDB","metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"psmdb.percona.com/v1\",\"kind\":\"PerconaServerMongoDB\",\"metadata\":{\"annotations\":{},\"finalizers\":[\"delete-psmdb-pods-in-order\"],\"name\":\"db-psmdb-db\",\"namespace\":\"percona\"},\"spec\":{\"allowUnsafeConfigurations\":false,\"backup\":{\"enabled\":false,\"image\":\"percona/percona-backup-mongodb:2.3.0\",\"serviceAccountName\":\"percona-server-mongodb-operator\"},\"crVersion\":\"1.15.0\",\"image\":\"percona/percona-server-mongodb:6.0.9-7\",\"imagePullPolicy\":\"Always\",\"pmm\":{\"enabled\":false,\"image\":\"percona/pmm-client:2.39.0\",\"serverHost\":\"monitoring-service\"},\"replsets\":[{\"affinity\":{\"antiAffinityTopologyKey\":\"kubernetes.io/hostname\"},\"arbiter\":{\"affinity\":{\"antiAffinityTopologyKey\":\"kubernetes.io/hostname\"},\"enabled\":false,\"resources\":{\"limits\":{\"cpu\":\"300m\",\"memory\":\"0.5G\"},\"requests\":{\"cpu\":\"300m\",\"memory\":\"0.5G\"}},\"size\":1},\"expose\":{\"enabled\":true,\"exposeType\":\"ClusterIP\"},\"name\":\"rs0\",\"nonvoting\":{\"affinity\":{\"antiAffinityTopologyKey\":\"kubernetes.io/hostname\"},\"enabled\":false,\"podDisruptionBudget\":{\"maxUnavailable\":1},\"resources\":{\"limits\":{\"cpu\":\"300m\",\"memory\":\"0.5G\"},\"requests\":{\"cpu\":\"300m\",\"memory\":\"0.5G\"}},\"size\":3,\"volumeSpec\":{\"persistentVolumeClaim\":{\"resources\":{\"requests\":{\"storage\":\"3Gi\"}}}}},\"podDisruptionBudget\":{\"maxUnavailable\":1},\"resources\":{\"limits\":{\"cpu\":\"300m\",\"memory\":\"0.5G\"},\"requests\":{\"cpu\":\"300m\",\"memory\":\"0.5G\"}},\"size\":3,\"splitHorizons\":{\"db-psmdb-db-rs0-0\":{\"external\":\"rs0-0.mycluster.xyz:27017\"},\"db-psmdb-db-rs0-1\":{\"external\":\"rs0-1.mycluster.xyz:27017\"},\"db-psmdb-db-rs0-2\":{\"external\":\"rs0-2.mycluster.xyz:27017\"}},\"volumeSpec\":{\"persistentVolumeClaim\":{\"resources\":{\"requests\":{\"storage\":\"3Gi\"}}}}}],\"secrets\":{\"encryptionKey\":\"db-psmdb-db-mongodb-encryption-key\",\"users\":\"db-psmdb-db-secrets\"},\"sharding\":{\"configsvrReplSet\":{\"affinity\":{\"antiAffinityTopologyKey\":\"kubernetes.io/hostname\"},\"expose\":{\"enabled\":false,\"exposeType\":\"ClusterIP\"},\"podDisruptionBudget\":{\"maxUnavailable\":1},\"resources\":{\"limits\":{\"cpu\":\"300m\",\"memory\":\"0.5G\"},\"requests\":{\"cpu\":\"300m\",\"memory\":\"0.5G\"}},\"size\":3,\"volumeSpec\":{\"persistentVolumeClaim\":{\"resources\":{\"requests\":{\"storage\":\"3Gi\"}}}}},\"enabled\":false,\"mongos\":{\"affinity\":{\"antiAffinityTopologyKey\":\"kubernetes.io/hostname\"},\"expose\":{\"exposeType\":\"ClusterIP\"},\"podDisruptionBudget\":{\"maxUnavailable\":1},\"resources\":{\"limits\":{\"cpu\":\"300m\",\"memory\":\"0.5G\"},\"requests\":{\"cpu\":\"300m\",\"memory\":\"0.5G\"}},\"size\":3}},\"updateStrategy\":\"SmartUpdate\",\"upgradeOptions\":{\"apply\":\"disabled\",\"schedule\":\"0 2 * * *\",\"setFCV\":false,\"versionServiceEndpoint\":\"https://check.percona.com\"}}}\n"},"creationTimestamp":"2023-10-13T20:59:53Z","finalizers":["delete-psmdb-pods-in-order"],"generation":1,"managedFields":[{"apiVersion":"psmdb.percona.com/v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:kubectl.kubernetes.io/last-applied-configuration":{}},"f:finalizers":{".":{},"v:\"delete-psmdb-pods-in-order\"":{}}},"f:spec":{".":{},"f:allowUnsafeConfigurations":{},"f:backup":{".":{},"f:enabled":{},"f:image":{},"f:serviceAccountName":{}},"f:crVersion":{},"f:image":{},"f:imagePullPolicy":{},"f:pmm":{".":{},"f:enabled":{},"f:image":{},"f:serverHost":{}},"f:replsets":{},"f:secrets":{".":{},"f:encryptionKey":{},"f:users":{}},"f:sharding":{".":{},"f:configsvrReplSet":{".":{},"f:affinity":{".":{},"f:antiAffinityTopologyKey":{}},"f:expose":{".":{},"f:enabled":{},"f:exposeType":{}},"f:podDisruptionBudget":{".":{},"f:maxUnavailable":{}},"f:resources":{".":{},"f:limits":{".":{},"f:cpu":{},"f:memory":{}},"f:requests":{".":{},"f:cpu":{},"f:memory":{}}},"f:size":{},"f:volumeSpec":{".":{},"f:persistentVolumeClaim":{".":{},"f:resources":{".":{},"f:requests":{".":{},"f:storage":{}}}}}},"f:enabled":{},"f:mongos":{".":{},"f:affinity":{".":{},"f:antiAffinityTopologyKey":{}},"f:expose":{".":{},"f:exposeType":{}},"f:podDisruptionBudget":{".":{},"f:maxUnavailable":{}},"f:resources":{".":{},"f:limits":{".":{},"f:cpu":{},"f:memory":{}},"f:requests":{".":{},"f:cpu":{},"f:memory":{}}},"f:size":{}}},"f:updateStrategy":{},"f:upgradeOptions":{".":{},"f:apply":{},"f:schedule":{},"f:setFCV":{},"f:versionServiceEndpoint":{}}}},"manager":"kubectl-client-side-apply","operation":"Update","time":"2023-10-13T20:59:53Z"},{"apiVersion":"psmdb.percona.com/v1","fieldsType":"FieldsV1","fieldsV1":{"f:status":{".":{},"f:conditions":{},"f:host":{},"f:mongoImage":{},"f:mongoVersion":{},"f:observedGeneration":{},"f:ready":{},"f:replsets":{".":{},"f:rs0":{".":{},"f:initialized":{},"f:ready":{},"f:size":{},"f:status":{}}},"f:size":{},"f:state":{}}},"manager":"percona-server-mongodb-operator","operation":"Update","subresource":"status","time":"2023-10-13T21:01:14Z"}],"name":"db-psmdb-db","namespace":"percona","resourceVersion":"1210776","uid":"796b42d8-4621-4e56-9de2-4966d14f7eaf"},"spec":{"allowUnsafeConfigurations":false,"backup":{"enabled":false,"image":"percona/percona-backup-mongodb:2.3.0","serviceAccountName":"percona-server-mongodb-operator"},"crVersion":"1.15.0","image":"percona/percona-server-mongodb:6.0.9-7","imagePullPolicy":"Always","pmm":{"enabled":false,"image":"percona/pmm-client:2.39.0","serverHost":"monitoring-service"},"replsets":[{"affinity":{"antiAffinityTopologyKey":"kubernetes.io/hostname"},"arbiter":{"affinity":{"antiAffinityTopologyKey":"kubernetes.io/hostname"},"enabled":false,"resources":{"limits":{"cpu":"300m","memory":"0.5G"},"requests":{"cpu":"300m","memory":"0.5G"}},"size":1},"expose":{"enabled":true,"exposeType":"ClusterIP"},"name":"rs0","nonvoting":{"affinity":{"antiAffinityTopologyKey":"kubernetes.io/hostname"},"enabled":false,"podDisruptionBudget":{"maxUnavailable":1},"resources":{"limits":{"cpu":"300m","memory":"0.5G"},"requests":{"cpu":"300m","memory":"0.5G"}},"size":3,"volumeSpec":{"persistentVolumeClaim":{"resources":{"requests":{"storage":"3Gi"}}}}},"podDisruptionBudget":{"maxUnavailable":1},"resources":{"limits":{"cpu":"300m","memory":"0.5G"},"requests":{"cpu":"300m","memory":"0.5G"}},"size":3,"splitHorizons":{"db-psmdb-db-rs0-0":{"external":"rs0-0.mycluster.xyz:27017"},"db-psmdb-db-rs0-1":{"external":"rs0-1.mycluster.xyz:27017"},"db-psmdb-db-rs0-2":{"external":"rs0-2.mycluster.xyz:27017"}},"volumeSpec":{"persistentVolumeClaim":{"resources":{"requests":{"storage":"3Gi"}}}}}],"secrets":{"encryptionKey":"db-psmdb-db-mongodb-encryption-key","users":"db-psmdb-db-secrets"},"sharding":{"configsvrReplSet":{"affinity":{"antiAffinityTopologyKey":"kubernetes.io/hostname"},"expose":{"enabled":false,"exposeType":"ClusterIP"},"podDisruptionBudget":{"maxUnavailable":1},"resources":{"limits":{"cpu":"300m","memory":"0.5G"},"requests":{"cpu":"300m","memory":"0.5G"}},"size":3,"volumeSpec":{"persistentVolumeClaim":{"resources":{"requests":{"storage":"3Gi"}}}}},"enabled":false,"mongos":{"affinity":{"antiAffinityTopologyKey":"kubernetes.io/hostname"},"expose":{"exposeType":"ClusterIP"},"podDisruptionBudget":{"maxUnavailable":1},"resources":{"limits":{"cpu":"300m","memory":"0.5G"},"requests":{"cpu":"300m","memory":"0.5G"}},"size":3}},"updateStrategy":"SmartUpdate","upgradeOptions":{"apply":"disabled","schedule":"0 2 * * *","setFCV":false,"versionServiceEndpoint":"https://check.percona.com"}},"status":{"conditions":[{"lastTransitionTime":"2023-10-13T20:59:56Z","status":"True","type":"initializing"},{"lastTransitionTime":"2023-10-13T21:01:08Z","message":"rs0: ready","reason":"RSReady","status":"True","type":"ready"},{"lastTransitionTime":"2023-10-13T21:01:08Z","status":"True","type":"initializing"},{"lastTransitionTime":"2023-10-13T21:01:14Z","status":"True","type":"ready"}],"host":"db-psmdb-db-rs0.percona.svc.cluster.local","mongoImage":"percona/percona-server-mongodb:6.0.9-7","mongoVersion":"6.0.9-7","observedGeneration":1,"ready":3,"replsets":{"rs0":{"initialized":true,"ready":3,"size":3,"status":"ready"}},"size":3,"state":"ready"}}”: strict decoding error: unknown field “spec.replsets[0].splitHorizons”

Tomislav_Plavcic · October 14, 2023, 9:22am

Hi @Louis_Laborde !

Because of validation failing I have a feeling that you have CRD from previous version installed in the system and helm when “psmdb-operator” chart is installed installs CRD only when it doesn’t exist, but helm doesn’t upgrade CRDs (it’s how helm works).

So please make sure you have CRD from version 1.15.0 installed, you can install it like this:
kubectl apply -f https://raw.githubusercontent.com/percona/percona-server-mongodb-operator/v1.15.0/deploy/crd.yaml --server-side

Then you can check if splitHorizons is available in your system CRD like this:

$ kubectl get crd perconaservermongodbs.psmdb.percona.com -oyaml|grep "splitHorizons" 
                    splitHorizons:
                      splitHorizons:

Also one other thing, make sure that you have crVersion field in your cr.yaml or helm values for psmdb-db chart set to 1.15.0.

Louis_Laborde · October 14, 2023, 2:36pm

Hello @Tomislav_Plavcic,
That was indeed the problem and my CRDs had not been updated by helm.
Thank you so much for your help.

An issue I see with Percona’s current implementation of this “splitHorizons” feature, besides the fact that it currently requires 2 steps (install without horizons followed by update with horizons) is that the port is hardcoded to 27017. Any port should be allowed to be specified by the user, and only default to 27017 if none is entered.

Tomislav_Plavcic · October 14, 2023, 2:57pm

Thanks for confirming!
If you wish you can open an improvement ticket for the fixed port issue or I will open it next week (need to check also how Mongo handles it).
Our Jira is at https://jira.percona.com/ and the correct project ID for PSMDB operator is K8SPSMDB.

Louis_Laborde · October 16, 2023, 2:58am

Hello @Tomislav_Plavcic,

Although the “horizons” field now shows up in rs.conf(), I am still not sure I understand what values the ports should have, and I still can’t access my replicas from outside the cluster.

When experimenting with MongoDB Community operator, the only way I was able to access my replicas from outside the cluster was by having a NodePort service for each pod of the replica AND using the same NodePort ports in my replsets Horizons.

To translate that to Percona MongoDB Operator, I believe that when requesting to expose the replica pods with NodePort (spec.replsets[0].expose.enabled=true AND spec.replsets[0].expose.exposeType=ClusterIP), there should be a way to pass the NodePort values (with default values automatically selected from available NodePorts) AND the Opertor should set the horizon ports to these same values.

For other service type like ClusterIP or LoadBalancer, I have no idea what port to put in the horizons.

I tried with MongoDB Community and Percona Operator to use traefik loadbalancer (default on K3s), but could not get it to work.

Topic		Replies	Views
splitHorizon DNS? Percona Operator for MongoDB	13	1794	February 22, 2024
Helm Upgrade from 1.11 to 1.12 confusing crds Percona Operator for MongoDB	5	824	August 18, 2022
Basic cluster with TLS not working, operator k8s/helm setup Percona Server for MongoDB mongodb , psmdb-operator	18	2949	December 18, 2022
Percona MongoDB operator backup failure Percona Operator for MongoDB percona , mongodb	4	1226	October 18, 2022
Percona MonogDB Cluster Fails when enabling TLS Percona Operator for MongoDB	3	182	June 5, 2024