Difficulty Creating Users - Authentication Errors

MongoDB Percona Operator for MongoDB
1

We’re just testing out using MongoDB in Kubernetes as a replacement for our existing platforms. Really at the moment we’re just doing throughput testing - we’re currently using AWS DocumentDB which seems prone to random slowdowns and is generally not particularly stable.

Percona seems to be a great solution. We’ve deployed the app using the EKS quickstart, and all seems to be fine. We’ve exposed the database via a Load Balancer and can connect remotely via the userAdmin account that’s created by default.

However, I’m having a problem that I’m really struggling to diagnose - whenever I create a new user within Mongo, I can connect to the DB from a local tty session, but that same user can’t connect remotely. I’m really struggling to diagnose this, as the userAdmin user can log in with no problems. Here’s the db.createUser command I’m using:

db.createUser({
    user: "george",
    pwd: "mypassword",
    roles: [
      {db: "admin", role: "userAdminAnyDatabase"},
    ]
})

When connecting via a remote pymongo session, I get the following error:

OperationFailure: Authentication failed., full error: {'operationTime': Timestamp(1643624885, 1), 'ok': 0.0, 'errmsg': 'Authentication failed.', 'code': 18, 'codeName': 'AuthenticationFailed', '$gleStats': {'lastOpTime': Timestamp(0, 0), 'electionId': ObjectId('7fffffff0000000000000003')}, 'lastCommittedOpTime': Timestamp(1643624885, 1), '$configServerState': {'opTime': {'ts': Timestamp(1643624885, 1), 't': 3}}, '$clusterTime': {'clusterTime': Timestamp(1643624885, 1), 'signature': {'hash': b'\'\xab\xf9\xf4"\xe0\xa8\xfe\xfd\x9d\x9e\xeeb\xb4\xfe{\xec\xbcV\xc9', 'keyId': 7059142238353227778}}}

I’m really struggling to diagnose this. I’m not sure if it’s an issue with the setup of the server, a Mongo issue, or potentially something I’ve just overlooked.

1 Like
2

Hello @Geo ,

thank you for submitting this.

I cannot reproduce the issue you are facing.
What I did:

  • deployed mongodb with the operator. Default bundle and cr.yaml.
  • Created new user from userAdmin
  • Connected to mongodb with the new user from pymongo

Anything specific about your setup? How are you exposing mongo? Are you running it with sharding or without? Please share the cr.yaml (or values.yaml for helm).

1 Like
3

Hi @Sergey_Pronin , thanks for looking into this for me!

Interesting that you’re unable to replicate! My steps for deploying are pretty straightforward I think - I’m pretty new to Kubernetes though, so could easily be doing something wrong. Here are my steps:

  • Deployed mongodb with the operator, bundle and cr.yaml files below. The only real difference was under the “expose” heading (“enabled” set to true, “exposeType” set to “LoadBalancer”.
  • Used the following commands to log in locally, which worked fine:
kubectl run -i --rm --tty percona-client --image=percona/percona-server-mongodb:4.4.10-11 --restart=Never -- bash -il
mongo "mongodb://userAdmin:userAdminPassword@my-cluster-name-mongos.<namespace name>.svc.cluster.local/admin?ssl=false"
  • Created my user with the code in my previous message, user was created in the admin database
  • Still in the tty session, logged out of Mongo as userAdmin and logged in successfully as the new user
  • Back to my local machine, using PyMongo, I then connected using the following string. This worked without any issue, but creating a database with this user is not allowed (fails with “not authorized on to execute command…”)
c = pymongo.MongoClient("mongodb://userAdmin:<useradminpassword>@<loadbalancer1>.eu-west-2.elb.amazonaws.com:27017,<loadbalancer2>.eu-west-2.elb.amazonaws.com:27017,<loadbalancer3>.eu-west-2.elb.amazonaws.com:27017/admin?ssl=false")
c.server_info()
  • Then try the exact same command with my new user, which fails with Authentication Error (Code 18). Exact command:
c = pymongo.MongoClient("mongodb://george:mypassword@<loadbalancer1>.eu-west-2.elb.amazonaws.com:27017,<loadbalancer2>.eu-west-2.elb.amazonaws.com:27017,<loadbalancer3>.eu-west-2.elb.amazonaws.com:27017/admin?ssl=false")
c.server_info()

So tl;dr - I can connect both as userAdmin and george when connected via ssh and logging into mongo locally. However, only userAdmin can connect via PyMongo remotely.

bundle.yaml:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: perconaservermongodbs.psmdb.percona.com
spec:
  group: psmdb.percona.com
  names:
    kind: PerconaServerMongoDB
    listKind: PerconaServerMongoDBList
    plural: perconaservermongodbs
    singular: perconaservermongodb
    shortNames:
    - psmdb
  scope: Namespaced
  versions:
    - name: v1alpha1
      storage: false
      served: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              x-kubernetes-preserve-unknown-fields: true
            status:
              type: object
              x-kubernetes-preserve-unknown-fields: true
      additionalPrinterColumns:
      - name: ENDPOINT
        type: string
        jsonPath: .status.host
      - name: Status
        type: string
        jsonPath: .status.state
      - name: Age
        type: date
        jsonPath: .metadata.creationTimestamp
      subresources:
        status: {}
    - name: v1
      storage: false
      served: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              x-kubernetes-preserve-unknown-fields: true
            status:
              type: object
              x-kubernetes-preserve-unknown-fields: true
      additionalPrinterColumns:
      - name: ENDPOINT
        type: string
        jsonPath: .status.host
      - name: Status
        type: string
        jsonPath: .status.state
      - name: Age
        type: date
        jsonPath: .metadata.creationTimestamp
      subresources:
        status: {}
    - name: v1-1-0
      storage: false
      served: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              x-kubernetes-preserve-unknown-fields: true
            status:
              type: object
              x-kubernetes-preserve-unknown-fields: true
      additionalPrinterColumns:
      - name: ENDPOINT
        type: string
        jsonPath: .status.host
      - name: Status
        type: string
        jsonPath: .status.state
      - name: Age
        type: date
        jsonPath: .metadata.creationTimestamp
      subresources:
        status: {}
    - name: v1-2-0
      storage: false
      served: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              x-kubernetes-preserve-unknown-fields: true
            status:
              type: object
              x-kubernetes-preserve-unknown-fields: true
      additionalPrinterColumns:
      - name: ENDPOINT
        type: string
        jsonPath: .status.host
      - name: Status
        type: string
        jsonPath: .status.state
      - name: Age
        type: date
        jsonPath: .metadata.creationTimestamp
      subresources:
        status: {}
    - name: v1-3-0
      storage: false
      served: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              x-kubernetes-preserve-unknown-fields: true
            status:
              type: object
              x-kubernetes-preserve-unknown-fields: true
      additionalPrinterColumns:
      - name: ENDPOINT
        type: string
        jsonPath: .status.host
      - name: Status
        type: string
        jsonPath: .status.state
      - name: Age
        type: date
        jsonPath: .metadata.creationTimestamp
      subresources:
        status: {}
    - name: v1-4-0
      storage: false
      served: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              x-kubernetes-preserve-unknown-fields: true
            status:
              type: object
              x-kubernetes-preserve-unknown-fields: true
      additionalPrinterColumns:
      - name: ENDPOINT
        type: string
        jsonPath: .status.host
      - name: Status
        type: string
        jsonPath: .status.state
      - name: Age
        type: date
        jsonPath: .metadata.creationTimestamp
      subresources:
        status: {}
    - name: v1-5-0
      storage: false
      served: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              x-kubernetes-preserve-unknown-fields: true
            status:
              type: object
              x-kubernetes-preserve-unknown-fields: true
      additionalPrinterColumns:
      - name: ENDPOINT
        type: string
        jsonPath: .status.host
      - name: Status
        type: string
        jsonPath: .status.state
      - name: Age
        type: date
        jsonPath: .metadata.creationTimestamp
      subresources:
        status: {}
    - name: v1-6-0
      storage: false
      served: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              x-kubernetes-preserve-unknown-fields: true
            status:
              type: object
              x-kubernetes-preserve-unknown-fields: true
      additionalPrinterColumns:
      - name: ENDPOINT
        type: string
        jsonPath: .status.host
      - name: Status
        type: string
        jsonPath: .status.state
      - name: Age
        type: date
        jsonPath: .metadata.creationTimestamp
      subresources:
        status: {}
    - name: v1-7-0
      storage: false
      served: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              x-kubernetes-preserve-unknown-fields: true
            status:
              type: object
              x-kubernetes-preserve-unknown-fields: true
      additionalPrinterColumns:
      - name: ENDPOINT
        type: string
        jsonPath: .status.host
      - name: Status
        type: string
        jsonPath: .status.state
      - name: Age
        type: date
        jsonPath: .metadata.creationTimestamp
      subresources:
        status: {}
    - name: v1-8-0
      storage: false
      served: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              x-kubernetes-preserve-unknown-fields: true
            status:
              type: object
              x-kubernetes-preserve-unknown-fields: true
      additionalPrinterColumns:
      - name: ENDPOINT
        type: string
        jsonPath: .status.host
      - name: Status
        type: string
        jsonPath: .status.state
      - name: Age
        type: date
        jsonPath: .metadata.creationTimestamp
      subresources:
        status: {}
    - name: v1-9-0
      storage: false
      served: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              x-kubernetes-preserve-unknown-fields: true
            status:
              type: object
              x-kubernetes-preserve-unknown-fields: true
      additionalPrinterColumns:
      - name: ENDPOINT
        type: string
        jsonPath: .status.host
      - name: Status
        type: string
        jsonPath: .status.state
      - name: Age
        type: date
        jsonPath: .metadata.creationTimestamp
      subresources:
        status: {}
    - name: v1-10-0
      storage: false
      served: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              x-kubernetes-preserve-unknown-fields: true
            status:
              type: object
              x-kubernetes-preserve-unknown-fields: true
      additionalPrinterColumns:
      - name: ENDPOINT
        type: string
        jsonPath: .status.host
      - name: Status
        type: string
        jsonPath: .status.state
      - name: Age
        type: date
        jsonPath: .metadata.creationTimestamp
      subresources:
        status: {}
    - name: v1-11-0
      storage: true
      served: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              x-kubernetes-preserve-unknown-fields: true
            status:
              type: object
              x-kubernetes-preserve-unknown-fields: true
      additionalPrinterColumns:
      - name: ENDPOINT
        type: string
        jsonPath: .status.host
      - name: Status
        type: string
        jsonPath: .status.state
      - name: Age
        type: date
        jsonPath: .metadata.creationTimestamp
      subresources:
        status: {}
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: perconaservermongodbbackups.psmdb.percona.com
spec:
  group: psmdb.percona.com
  names:
    kind: PerconaServerMongoDBBackup
    listKind: PerconaServerMongoDBBackupList
    plural: perconaservermongodbbackups
    singular: perconaservermongodbbackup
    shortNames:
    - psmdb-backup
  scope: Namespaced
  versions:
    - name: v1
      storage: true
      served: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              x-kubernetes-preserve-unknown-fields: true
            status:
              type: object
              x-kubernetes-preserve-unknown-fields: true
      additionalPrinterColumns:
        - name: Cluster
          type: string
          description: Cluster name
          jsonPath: .spec.psmdbCluster
        - name: Storage
          type: string
          description: Storage name from pxc spec
          jsonPath: .spec.storageName
        - name: Destination
          type: string
          description: Backup destination
          jsonPath: .status.destination
        - name: Status
          type: string
          description: Job status
          jsonPath: .status.state
        - name: Completed
          description: Completed time
          type: date
          jsonPath: .status.completed
        - name: Age
          type: date
          jsonPath: .metadata.creationTimestamp
      subresources:
        status: {}
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: perconaservermongodbrestores.psmdb.percona.com
spec:
  group: psmdb.percona.com
  names:
    kind: PerconaServerMongoDBRestore
    listKind: PerconaServerMongoDBRestoreList
    plural: perconaservermongodbrestores
    singular: perconaservermongodbrestore
    shortNames:
    - psmdb-restore
  scope: Namespaced
  versions:
    - name: v1
      storage: true
      served: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              x-kubernetes-preserve-unknown-fields: true
            status:
              type: object
              x-kubernetes-preserve-unknown-fields: true
      additionalPrinterColumns:
        - name: Cluster
          type: string
          description: Cluster name
          jsonPath: .spec.clusterName
        - name: Status
          type: string
          description: Job status
          jsonPath: .status.state
        - name: Age
          type: date
          jsonPath: .metadata.creationTimestamp
      subresources:
        status: {}
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: percona-server-mongodb-operator
rules:
- apiGroups:
  - psmdb.percona.com
  resources:
  - perconaservermongodbs
  - perconaservermongodbs/status
  - perconaservermongodbbackups
  - perconaservermongodbbackups/status
  - perconaservermongodbrestores
  - perconaservermongodbrestores/status
  verbs:
  - get
  - list
  - watch
  - create
  - update
  - patch
  - delete
- apiGroups:
  - ""
  resources:
  - pods
  - pods/exec
  - services
  - persistentvolumeclaims
  - secrets
  - configmaps
  verbs:
  - get
  - list
  - watch
  - create
  - update
  - patch
  - delete
- apiGroups:
  - apps
  resources:
  - deployments
  - replicasets
  - statefulsets
  verbs:
  - get
  - list
  - watch
  - create
  - update
  - patch
  - delete
- apiGroups:
  - batch
  resources:
  - cronjobs
  verbs:
  - get
  - list
  - watch
  - create
  - update
  - patch
  - delete
- apiGroups:
  - policy
  resources:
  - poddisruptionbudgets
  verbs:
  - get
  - list
  - watch
  - create
  - update
  - patch
  - delete
- apiGroups:
  - certmanager.k8s.io
  - cert-manager.io
  resources:
  - issuers
  - certificates
  verbs:
  - get
  - list
  - watch
  - create
  - update
  - patch
  - delete
  - deletecollection
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: percona-server-mongodb-operator
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: service-account-percona-server-mongodb-operator
subjects:
- kind: ServiceAccount
  name: percona-server-mongodb-operator
roleRef:
  kind: Role
  name: percona-server-mongodb-operator
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: percona-server-mongodb-operator
spec:
  replicas: 1
  selector:
    matchLabels:
      name: percona-server-mongodb-operator
  template:
    metadata:
      labels:
        name: percona-server-mongodb-operator
    spec:
      serviceAccountName: percona-server-mongodb-operator
      containers:
        - name: percona-server-mongodb-operator
          image: percona/percona-server-mongodb-operator:1.11.0
          ports:
          - containerPort: 60000
            name: metrics
          command:
          - percona-server-mongodb-operator
          imagePullPolicy: Always
          env:
            - name: WATCH_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: OPERATOR_NAME
              value: percona-server-mongodb-operator
            - name: RESYNC_PERIOD
              value: 5s
            - name: LOG_VERBOSE
              value: "false"

cr.yaml

apiVersion: psmdb.percona.com/v1-11-0
kind: PerconaServerMongoDB
metadata:
  name: my-cluster-name
#  finalizers:
#    - delete-psmdb-pvc
spec:
#  platform: openshift
#  clusterServiceDNSSuffix: svc.cluster.local
#  pause: true
#  unmanaged: false
  crVersion: 1.11.0
  image: percona/percona-server-mongodb:4.4.10-11
  imagePullPolicy: Always
#  imagePullSecrets:
#    - name: private-registry-credentials
#  runUid: 1001
  allowUnsafeConfigurations: false
  updateStrategy: SmartUpdate
  upgradeOptions:
    versionServiceEndpoint: https://check.percona.com
    apply: 4.4-recommended
    schedule: "0 2 * * *"
    setFCV: false
  secrets:
    users: my-cluster-name-secrets
  pmm:
    enabled: false
    image: percona/pmm-client:2.24.0
    serverHost: monitoring-service
#    mongodParams: --environment=ENVIRONMENT
#    mongosParams: --environment=ENVIRONMENT
  replsets:

  - name: rs0
    size: 3
#    externalNodes:
#    - host: 34.124.76.90
#    - host: 34.124.76.91
#      port: 27017
#      votes: 0
#      priority: 0
#    - host: 34.124.76.92
#    # for more configuration fields refer to https://docs.mongodb.com/manual/reference/configuration-options/
#    configuration: |
#      operationProfiling:
#        mode: slowOp
#      systemLog:
#        verbosity: 1
#      storage:
#        engine: wiredTiger
#        inMemory:
#          engineConfig:
#            inMemorySizeRatio: 0.9
#        wiredTiger:
#          engineConfig:
#            cacheSizeRatio: 0.5
#            directoryForIndexes: false
#            journalCompressor: snappy
#          collectionConfig:
#            blockCompressor: snappy
#          indexConfig:
#            prefixCompression: true
    affinity:
      antiAffinityTopologyKey: "kubernetes.io/hostname"
#      advanced:
#        nodeAffinity:
#          requiredDuringSchedulingIgnoredDuringExecution:
#            nodeSelectorTerms:
#            - matchExpressions:
#              - key: kubernetes.io/e2e-az-name
#                operator: In
#                values:
#                - e2e-az1
#                - e2e-az2
#    tolerations:
#    - key: "node.alpha.kubernetes.io/unreachable"
#      operator: "Exists"
#      effect: "NoExecute"
#      tolerationSeconds: 6000
#    priorityClassName: high-priority
#    annotations:
#      iam.amazonaws.com/role: role-arn
#    labels:
#      rack: rack-22
#    nodeSelector:
#      disktype: ssd
#    livenessProbe:
#      failureThreshold: 4
#      initialDelaySeconds: 60
#      periodSeconds: 30
#      timeoutSeconds: 10
#      startupDelaySeconds: 7200
#    readinessProbe:
#      failureThreshold: 8
#      initialDelaySeconds: 10
#      periodSeconds: 3
#      successThreshold: 1
#      timeoutSeconds: 2
#    runtimeClassName: image-rc
#    sidecars:
#    - image: busybox
#      command: ["/bin/sh"]
#      args: ["-c", "while true; do echo echo $(date -u) 'test' >> /dev/null; sleep 5;done"]
#      name: rs-sidecar-1
#      volumeMounts:
#        - mountPath: /volume1
#          name: sidecar-volume-claim
#        - mountPath: /secret
#          name: sidecar-secret
#        - mountPath: /configmap
#          name: sidecar-config
#    sidecarVolumes:
#    - name: sidecar-secret
#      secret:
#        secretName: mysecret
#    - name: sidecar-config
#      configMap:
#        name: myconfigmap
#    sidecarPVCs:
#    - apiVersion: v1
#      kind: PersistentVolumeClaim
#      metadata:
#        name: sidecar-volume-claim
#      spec:
#        resources:
#          requests:
#            storage: 1Gi
#        volumeMode: Filesystem
#        accessModes:
#          - ReadWriteOnce
    podDisruptionBudget:
      maxUnavailable: 1
#      minAvailable: 0
    expose:
      enabled: true
      exposeType: LoadBalancer
#      loadBalancerSourceRanges:
#        - 10.0.0.0/8
#      serviceAnnotations:
#        service.beta.kubernetes.io/aws-load-balancer-backend-protocol: http
    nonvoting:
      enabled: false
      size: 3
#      # for more configuration fields refer to https://docs.mongodb.com/manual/reference/configuration-options/
#      configuration: |
#        operationProfiling:
#          mode: slowOp
#        systemLog:
#          verbosity: 1
      affinity:
        antiAffinityTopologyKey: "kubernetes.io/hostname"
#        advanced:
#          nodeAffinity:
#            requiredDuringSchedulingIgnoredDuringExecution:
#              nodeSelectorTerms:
#              - matchExpressions:
#                - key: kubernetes.io/e2e-az-name
#                  operator: In
#                  values:
#                  - e2e-az1
#                  - e2e-az2
#      tolerations:
#      - key: "node.alpha.kubernetes.io/unreachable"
#        operator: "Exists"
#        effect: "NoExecute"
#        tolerationSeconds: 6000
#      priorityClassName: high-priority
#      annotations:
#        iam.amazonaws.com/role: role-arn
#      labels:
#        rack: rack-22
#      nodeSelector:
#        disktype: ssd
#    schedulerName: "default"
      podDisruptionBudget:
        maxUnavailable: 1
#        minAvailable: 0
      resources:
        limits:
          cpu: "300m"
          memory: "0.5G"
        requests:
          cpu: "300m"
          memory: "0.5G"
      volumeSpec:
#        emptyDir: {}
#        hostPath:
#          path: /data
#          type: Directory
        persistentVolumeClaim:
#          storageClassName: standard
#          accessModes: [ "ReadWriteOnce" ]
          resources:
            requests:
              storage: 1Gi
    arbiter:
      enabled: false
      size: 1
      affinity:
        antiAffinityTopologyKey: "kubernetes.io/hostname"
#        advanced:
#          nodeAffinity:
#            requiredDuringSchedulingIgnoredDuringExecution:
#              nodeSelectorTerms:
#              - matchExpressions:
#                - key: kubernetes.io/e2e-az-name
#                  operator: In
#                  values:
#                  - e2e-az1
#                  - e2e-az2
#      tolerations:
#      - key: "node.alpha.kubernetes.io/unreachable"
#        operator: "Exists"
#        effect: "NoExecute"
#        tolerationSeconds: 6000
#      priorityClassName: high-priority
#      annotations:
#        iam.amazonaws.com/role: role-arn
#      labels:
#        rack: rack-22
#      nodeSelector:
#        disktype: ssd
#    schedulerName: "default"
    resources:
      limits:
        cpu: "300m"
        memory: "0.5G"
      requests:
        cpu: "300m"
        memory: "0.5G"
    volumeSpec:
#      emptyDir: {}
#      hostPath:
#        path: /data
#        type: Directory
      persistentVolumeClaim:
#        storageClassName: standard
#        accessModes: [ "ReadWriteOnce" ]
        resources:
          requests:
            storage: 3Gi

  sharding:
    enabled: true

    configsvrReplSet:
      size: 3
#      externalNodes:
#      - host: 34.124.76.93
#      - host: 34.124.76.94
#        port: 27017
#        votes: 0
#        priority: 0
#      - host: 34.124.76.95
#      # for more configuration fields refer to https://docs.mongodb.com/manual/reference/configuration-options/
#      configuration: |
#        operationProfiling:
#          mode: slowOp
#        systemLog:
#           verbosity: 1
      affinity:
        antiAffinityTopologyKey: "kubernetes.io/hostname"
#        advanced:
#          nodeAffinity:
#            requiredDuringSchedulingIgnoredDuringExecution:
#              nodeSelectorTerms:
#              - matchExpressions:
#                - key: kubernetes.io/e2e-az-name
#                  operator: In
#                  values:
#                  - e2e-az1
#                  - e2e-az2
#      tolerations:
#      - key: "node.alpha.kubernetes.io/unreachable"
#        operator: "Exists"
#        effect: "NoExecute"
#        tolerationSeconds: 6000
#      priorityClassName: high-priority
#      annotations:
#        iam.amazonaws.com/role: role-arn
#      labels:
#        rack: rack-22
#      nodeSelector:
#        disktype: ssd
#      storage:
#        engine: wiredTiger
#        wiredTiger:
#          engineConfig:
#            cacheSizeRatio: 0.5
#            directoryForIndexes: false
#            journalCompressor: snappy
#          collectionConfig:
#            blockCompressor: snappy
#          indexConfig:
#            prefixCompression: true
#      livenessProbe:
#        failureThreshold: 4
#        initialDelaySeconds: 60
#        periodSeconds: 30
#        timeoutSeconds: 10
#        startupDelaySeconds: 7200
#      readinessProbe:
#        failureThreshold: 3
#        initialDelaySeconds: 10
#        periodSeconds: 3
#        successThreshold: 1
#        timeoutSeconds: 2
#      runtimeClassName: image-rc
#      sidecars:
#      - image: busybox
#        command: ["/bin/sh"]
#        args: ["-c", "while true; do echo echo $(date -u) 'test' >> /dev/null; sleep 5;done"]
#        name: rs-sidecar-1
      podDisruptionBudget:
        maxUnavailable: 1
      expose:
        enabled: false
        exposeType: ClusterIP
#        loadBalancerSourceRanges:
#          - 10.0.0.0/8
#        serviceAnnotations:
#          service.beta.kubernetes.io/aws-load-balancer-backend-protocol: http
      resources:
        limits:
          cpu: "300m"
          memory: "0.5G"
        requests:
          cpu: "300m"
          memory: "0.5G"
      volumeSpec:
#       emptyDir: {}
#       hostPath:
#         path: /data
#         type: Directory
        persistentVolumeClaim:
#          storageClassName: standard
#          accessModes: [ "ReadWriteOnce" ]
          resources:
            requests:
              storage: 3Gi

    mongos:
      size: 3
#      # for more configuration fields refer to https://docs.mongodb.com/manual/reference/configuration-options/
#      configuration: |
#        systemLog:
#           verbosity: 1
      affinity:
        antiAffinityTopologyKey: "kubernetes.io/hostname"
#        advanced:
#          nodeAffinity:
#            requiredDuringSchedulingIgnoredDuringExecution:
#              nodeSelectorTerms:
#              - matchExpressions:
#                - key: kubernetes.io/e2e-az-name
#                  operator: In
#                  values:
#                  - e2e-az1
#                  - e2e-az2
#      tolerations:
#      - key: "node.alpha.kubernetes.io/unreachable"
#        operator: "Exists"
#        effect: "NoExecute"
#        tolerationSeconds: 6000
#      priorityClassName: high-priority
#      annotations:
#        iam.amazonaws.com/role: role-arn
#      labels:
#        rack: rack-22
#      nodeSelector:
#        disktype: ssd
#      livenessProbe:
#        failureThreshold: 4
#        initialDelaySeconds: 60
#        periodSeconds: 30
#        timeoutSeconds: 10
#        startupDelaySeconds: 7200
#      readinessProbe:
#        failureThreshold: 3
#        initialDelaySeconds: 10
#        periodSeconds: 3
#        successThreshold: 1
#        timeoutSeconds: 2
#      runtimeClassName: image-rc
#      sidecars:
#      - image: busybox
#        command: ["/bin/sh"]
#        args: ["-c", "while true; do echo echo $(date -u) 'test' >> /dev/null; sleep 5;done"]
#        name: rs-sidecar-1
      podDisruptionBudget:
        maxUnavailable: 1
      resources:
        limits:
          cpu: "300m"
          memory: "0.5G"
        requests:
          cpu: "300m"
          memory: "0.5G"
      expose:
        exposeType: ClusterIP
#        loadBalancerSourceRanges:
#          - 10.0.0.0/8
#        serviceAnnotations:
#          service.beta.kubernetes.io/aws-load-balancer-backend-protocol: http
#      auditLog:
#        destination: file
#        format: BSON
#        filter: '{}'

  mongod:
    net:
      port: 27017
      hostPort: 0
    security:
      redactClientLogData: false
      enableEncryption: true
      encryptionKeySecret: my-cluster-name-mongodb-encryption-key
      encryptionCipherMode: AES256-CBC
    setParameter:
      ttlMonitorSleepSecs: 60
      wiredTigerConcurrentReadTransactions: 128
      wiredTigerConcurrentWriteTransactions: 128
    storage:
      engine: wiredTiger
      inMemory:
        engineConfig:
          inMemorySizeRatio: 0.9
      wiredTiger:
        engineConfig:
          cacheSizeRatio: 0.5
          directoryForIndexes: false
          journalCompressor: snappy
        collectionConfig:
          blockCompressor: snappy
        indexConfig:
          prefixCompression: true
    operationProfiling:
      mode: slowOp
      slowOpThresholdMs: 100
      rateLimit: 100
#    auditLog:
#      destination: file
#      format: BSON
#      filter: '{}'

  backup:
    enabled: true
    restartOnFailure: true
    image: percona/percona-server-mongodb-operator:1.11.0-backup
    serviceAccountName: percona-server-mongodb-operator
#    annotations:
#      iam.amazonaws.com/role: role-arn
#    resources:
#      limits:
#        cpu: "300m"
#        memory: "0.5G"
#      requests:
#        cpu: "300m"
#        memory: "0.5G"
    storages:
#      s3-us-west:
#        type: s3
#        s3:
#          bucket: S3-BACKUP-BUCKET-NAME-HERE
#          credentialsSecret: my-cluster-name-backup-s3
#          region: us-west-2
#          prefix: ""
#      minio:
#        type: s3
#        s3:
#          bucket: MINIO-BACKUP-BUCKET-NAME-HERE
#          region: us-east-1
#          credentialsSecret: my-cluster-name-backup-minio
#          endpointUrl: http://minio.psmdb.svc.cluster.local:9000/minio/
#          prefix: ""
#      azure-blob:
#        type: azure
#        azure:
#          container: CONTAINER-NAME
#          prefix: PREFIX-NAME
#          credentialsSecret: SECRET-NAME
    pitr:
      enabled: false
#      oplogSpanMin: 10
    tasks:
#      - name: daily-s3-us-west
#        enabled: true
#        schedule: "0 0 * * *"
#        keep: 3
#        storageName: s3-us-west
#        compressionType: gzip
#      - name: weekly-s3-us-west
#        enabled: false
#        schedule: "0 0 * * 0"
#        keep: 5
#        storageName: s3-us-west
#        compressionType: gzip
1 Like
4

@Geo got it now.

By default you have sharding enabled, so you cluster looks like this:

Main temporary - Page 1 (1)
Main temporary - Page 1 (1)860×520 13.7 KB

  1. you create the user connecting to mongos
  2. as a result the user is created on Config Server
  3. then you try to connect to the replica set directly (through load balancers) and it fails. Reason: replica set does not know about this user as mongos and config servers handle the authentication here.

There are two ways:

  1. You can connect with this user through mongos (as it should be). You don’t need to expose replica set itself. It is enough to expose mongos.
  2. If you don’t need sharding - you can disable it and expose replica set.

Let me know if it helps.

3 Likes
5

Aha perfect! I’ll have a go today and report back. Thanks for your help!

1 Like
6

Are you connecting via the mongos service (ok) or do you connect directly to a specific mongo (nok) ?

→ pls provide connection URL

1 Like