I have deployed the percona operator and MongoDB server on my kubernetes cluster. I am exposing the service externally via Loadbalancer. As of this new installation today i’m no longer able to externally connnect.
I get the following error:
MongoServerSelectionError: Server selection timed out after 30000 ms
mongosh --host k8s-mongodb-psmdbdbr-13dd3882fd-14dc163145fe98e1.elb.us-east-2.amazonaws.com:27017 --username databaseAdmin --authenticationDatabase admin
Enter password: *****************
Current Mongosh Log ID: 6454fdf41059a0d5989c031e
Connecting to: mongodb://<credentials>@k8s-mongodb-psmdbdbr-13dd3882fd-14dc163145fe98e1.elb.us-east-2.amazonaws.com:27017/?directConnection=true&authSource=admin&appName=mongosh+1.5.1
Am i doing something obviously wrong with this deployment, i seem to had this working a few days back. A second pair of eyes is much appreciated. thanks
kubectl get all -n mongodb
NAME READY STATUS RESTARTS AGE
pod/psmdb-db-rs0-0 3/3 Running 0 52m
pod/psmdb-operator-869b9b99d-slmbf 1/1 Running 0 56m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/psmdb-db-rs0 ClusterIP None <none> 27017/TCP 52m
service/psmdb-db-rs0-0 LoadBalancer 172.20.148.113 k8s-mongodb-psmdbdbr-13dd3882fd-14dc163145fe98e1.elb.us-east-2.amazonaws.com 27017:32643/TCP 51m
service/psmdb-metrics ClusterIP 172.20.43.105 <none> 9216/TCP 51m
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/psmdb-operator 1/1 1 1 56m
NAME DESIRED CURRENT READY AGE
replicaset.apps/psmdb-operator-869b9b99d 1 1 1 56m
NAME READY AGE
statefulset.apps/psmdb-db-rs0 1/1 52m
# Default values for psmdb-cluster.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.
# Platform type: kubernetes, openshift
# platform: kubernetes
# Cluster DNS Suffix
# clusterServiceDNSSuffix: svc.cluster.local
# clusterServiceDNSMode: "External"
finalizers:
## Set this if you want that operator deletes the primary pod last
- delete-psmdb-pods-in-order
## Set this if you want to delete database persistent volumes on cluster deletion
# - delete-psmdb-pvc
nameOverride: ""
fullnameOverride: ""
env:
name: LOG_STRUCTURED
value: 'false'
name: LOG_LEVEL
value: DEBUG
crVersion: 1.14.0
pause: false
unmanaged: false
allowUnsafeConfigurations: true
# ignoreAnnotations:
# - service.beta.kubernetes.io/aws-load-balancer-backend-protocol
# ignoreLabels:
# - rack
multiCluster:
enabled: false
# DNSSuffix: svc.clusterset.local
updateStrategy: SmartUpdate
upgradeOptions:
versionServiceEndpoint: https://check.percona.com
apply: disabled
schedule: "0 2 * * *"
setFCV: false
image:
repository: percona/percona-server-mongodb
tag: 6.0.4-3
imagePullPolicy: Always
# imagePullSecrets: []
# initImage:
# repository: percona/percona-server-mongodb-operator
# tag: 1.14.0
# initContainerSecurityContext: {}
# tls:
# # 90 days in hours
# certValidityDuration: 2160h
secrets: {}
# If you set users secret here the operator will use existing one or generate random values
# If not set the operator generates the default secret with name <cluster_name>-secrets
# users: my-cluster-name-secrets
# encryptionKey: my-cluster-name-mongodb-encryption-key
pmm:
enabled: false
image:
repository: percona/pmm-client
tag: 2.35.0
serverHost: monitoring-service
replsets:
- name: rs0
size: 1
# externalNodes:
# - host: 34.124.76.90
# - host: 34.124.76.91
# port: 27017
# votes: 0
# priority: 0
# - host: 34.124.76.92
# configuration: |
# operationProfiling:
# mode: slowOp
# systemLog:
# verbosity: 1
antiAffinityTopologyKey: "kubernetes.io/hostname"
# tolerations: []
# priorityClass: ""
# annotations: {}
# labels: {}
nodeSelector:
geeiq/node-type: "ops"
# livenessProbe:
# failureThreshold: 4
# initialDelaySeconds: 60
# periodSeconds: 30
# timeoutSeconds: 10
# startupDelaySeconds: 7200
# readinessProbe:
# failureThreshold: 8
# initialDelaySeconds: 10
# periodSeconds: 3
# successThreshold: 1
# timeoutSeconds: 2
# runtimeClassName: image-rc
# storage:
# engine: wiredTiger
# wiredTiger:
# engineConfig:
# cacheSizeRatio: 0.5
# directoryForIndexes: false
# journalCompressor: snappy
# collectionConfig:
# blockCompressor: snappy
# indexConfig:
# prefixCompression: true
# inMemory:
# engineConfig:
# inMemorySizeRatio: 0.5
sidecars:
- image: percona/mongodb_exporter:0.36
env:
- name: EXPORTER_USER
valueFrom:
secretKeyRef:
name: psmdb-db-secrets
key: MONGODB_CLUSTER_MONITOR_USER
- name: EXPORTER_PASS
valueFrom:
secretKeyRef:
name: psmdb-db-secrets
key: MONGODB_CLUSTER_MONITOR_PASSWORD
- name: POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
- name: MONGODB_URI
value: "mongodb://$(EXPORTER_USER):$(EXPORTER_PASS)@$(POD_IP):27017"
args: ["--discovering-mode", "--compatible-mode", "--collect-all", "--mongodb.uri=$(MONGODB_URI)"]
name: metrics
# volumeMounts:
# - mountPath: /volume1
# name: sidecar-volume-claim
# - mountPath: /secret
# name: sidecar-secret
# - mountPath: /configmap
# name: sidecar-config
# sidecarVolumes:
# - name: sidecar-secret
# secret:
# secretName: mysecret
# - name: sidecar-config
# configMap:
# name: myconfigmap
# sidecarPVCs:
# - apiVersion: v1
# kind: PersistentVolumeClaim
# metadata:
# name: sidecar-volume-claim
# spec:
# resources:
# requests:
# storage: 1Gi
# volumeMode: Filesystem
# accessModes:
# - ReadWriteOnce
podDisruptionBudget:
maxUnavailable: 1
expose:
enabled: true
exposeType: LoadBalancer
# loadBalancerSourceRanges:
# - 10.0.0.0/8
# serviceAnnotations:
# service.beta.kubernetes.io/aws-load-balancer-backend-protocol: http
# serviceLabels:
# some-label: some-key
nonvoting:
enabled: false
# podSecurityContext: {}
# containerSecurityContext: {}
size: 3
# configuration: |
# operationProfiling:
# mode: slowOp
# systemLog:
# verbosity: 1
antiAffinityTopologyKey: "kubernetes.io/hostname"
# tolerations: []
# priorityClass: ""
# annotations: {}
# labels: {}
# nodeSelector: {}
podDisruptionBudget:
maxUnavailable: 1
resources:
limits:
cpu: "300m"
memory: "0.5G"
requests:
cpu: "300m"
memory: "0.5G"
volumeSpec:
# emptyDir: {}
# hostPath:
# path: /data
pvc:
# annotations:
# volume.beta.kubernetes.io/storage-class: example-hostpath
# labels:
# rack: rack-22
# storageClassName: standard
# accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: 3Gi
arbiter:
enabled: false
size: 1
antiAffinityTopologyKey: "kubernetes.io/hostname"
# tolerations: []
# priorityClass: ""
# annotations: {}
# labels: {}
# nodeSelector: {}
# schedulerName: ""
# resources:
# limits:
# cpu: "300m"
# memory: "0.5G"
# requests:
# cpu: "300m"
# memory: "0.5G"
volumeSpec:
# emptyDir: {}
# hostPath:
# path: /data
pvc:
# annotations:
# volume.beta.kubernetes.io/storage-class: example-hostpath
# labels:
# rack: rack-22
storageClassName: mongodb
# accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: 250Gi
sharding:
enabled: false
backup:
enabled: true
image:
repository: percona/percona-backup-mongodb
tag: 2.0.5
serviceAccountName: percona-server-mongodb-operator
# annotations:
# iam.amazonaws.com/role: arn:aws:iam::700849607999:role/geeiq-test-default-eks-mongodb
# resources:
# limits:
# cpu: "300m"
# memory: "0.5G"
# requests:
# cpu: "300m"
# memory: "0.5G"
storages:
s3-us-east:
type: s3
s3:
bucket: geeiq-prod-mongodb-backup
credentialsSecret: prod-aws-mongodb
region: us-east-2
prefix: ""
uploadPartSize: 10485760
maxUploadParts: 10000
storageClass: STANDARD
insecureSkipTLSVerify: false
# minio:
# type: s3
# s3:
# bucket: MINIO-BACKUP-BUCKET-NAME-HERE
# region: us-east-1
# credentialsSecret: my-cluster-name-backup-minio
# endpointUrl: http://minio.psmdb.svc.cluster.local:9000/minio/
# prefix: ""
# azure-blob:
# type: azure
# azure:
# container: CONTAINER-NAME
# prefix: PREFIX-NAME
# credentialsSecret: SECRET-NAME
pitr:
enabled: false
# oplogSpanMin: 10
# compressionType: gzip
# compressionLevel: 6
tasks:
- name: "daily-s3-backup"
enabled: true
schedule: "0 1 * * *"
keep: 3
type: logical
storageName: s3-us-east
# - name: daily-s3-us-west
# enabled: true
# schedule: "0 0 * * *"
# keep: 3
# storageName: s3-us-west
# compressionType: gzip
# - name: weekly-s3-us-west
# enabled: false
# schedule: "0 0 * * 0"
# keep: 5
# storageName: s3-us-west
# compressionType: gzip
# - name: weekly-s3-us-west-physical
# enabled: false
# schedule: "0 5 * * 0"
# keep: 5
# type: physical
# storageName: s3-us-west
# compressionType: gzip
# compressionLevel: 6
# If you set users here the secret will be constructed by helm with these values
# users:
# MONGODB_BACKUP_USER: backup
# MONGODB_BACKUP_PASSWORD: backup123456
# MONGODB_DATABASE_ADMIN_USER: databaseAdmin
# MONGODB_DATABASE_ADMIN_PASSWORD: databaseAdmin123456
# MONGODB_CLUSTER_ADMIN_USER: clusterAdmin
# MONGODB_CLUSTER_ADMIN_PASSWORD: clusterAdmin123456
# MONGODB_CLUSTER_MONITOR_USER: clusterMonitor
# MONGODB_CLUSTER_MONITOR_PASSWORD: clusterMonitor123456
# MONGODB_USER_ADMIN_USER: userAdmin
# MONGODB_USER_ADMIN_PASSWORD: userAdmin123456
# PMM_SERVER_API_KEY: apikey
# # PMM_SERVER_USER: admin
# # PMM_SERVER_PASSWORD: admin
I dont remember the created LB being of type network previously, has that changed with an update recently?
