Can't do backups in K8s cluster

MongoDB Percona Backup for MongoDB
1

I’m trying to establish a PITR to Minio. Both replicated, no sharding.

These are the logs:

psmdb-db

    Last Transition Time:  2021-12-15T13:37:24Z
    Status:                True
    Type:                  ready
    Last Transition Time:  2021-12-15T13:38:46Z
    Message:               create pbm object: create PBM connection to percona-mongo-psmdb-d-rs0-2.percona-mongo-psmdb-d-rs0.default.svc.cluster.local:27017,percona-mongo-psmdb-d-rs0-0.percona-mongo-psmdb-d-rs0.default.svc.cluster.local:27017
: create mongo connection: mongo ping: server selection error: server selection timeout, current topology: { Type: Unknown, Servers: [{ Addr: percona-mongo-psmdb-d-rs0-2.percona-mongo-psmdb-d-rs0.default.svc.cluster.local:27017, Type: Unknown
, Last error: connection() error occured during connection handshake: dial tcp: lookup percona-mongo-psmdb-d-rs0-2.percona-mongo-psmdb-d-rs0.default.svc.cluster.local on 169.254.20.10:53: no such host }, { Addr: percona-mongo-psmdb-d-rs0-0.pe
rcona-mongo-psmdb-d-rs0.default.svc.cluster.local:27017, Type: RSGhost, Average RTT: 2332184 }, ] }
    Reason:                ErrorReconcile
    Status:                True
    Type:                  error
    Last Transition Time:  2021-12-15T13:38:47Z
    Status:                True
    Type:                  initializing
    Last Transition Time:  2021-12-15T13:39:37Z
    Message:               rs0: ready
    Reason:                RSReady
    Status:                True
    Type:                  ready
    Last Transition Time:  2021-12-15T13:39:37Z
    Status:                True
    Type:                  initializing
    Last Transition Time:  2021-12-15T13:39:42Z
    Status:                True
    Type:                  ready
  Host:                    percona-mongo-psmdb-d-rs0.default.svc.cluster.local
  Mongo Image:             percona/percona-server-mongodb:4.4.8-9
  Mongo Version:           4.4.8-9
  Observed Generation:     11
  Pmm Version:             2.21.0
  Ready:                   3
  Replsets:
    rs0:
      Initialized:  true
      Ready:        3
      Size:         3
      Status:       ready
  Size:             3
  State:            ready
Events:             <none>
psmdb-db $

spec.backup.storages

  minio:
      type: s3
      s3:
        bucket: bucket-1
        credentialsSecret: myskills-minio-bucket
        endpointUrl: https://minio.default.svc.cluster.local:443

#    fs-pvc:
#      type: filesystem
#      volume:
#        persistentVolumeClaim:
#          storageClassName: longhorn-static
#          accessModes: [ "ReadWriteOnce" ]
#          resources:
#            requests:
#              storage: 10Gi

  pitr:
    enabled: true
  tasks:
   - name: daily-psmdb-backup
     enabled: true
     schedule: "0 0 * * *"
     keep: 3
     storageName: minio
     compressionType: gzip

psmdb-backup

Name:         backup1
Namespace:    default
Labels:       <none>
Annotations:  <none>
API Version:  psmdb.percona.com/v1
Kind:         PerconaServerMongoDBBackup
Metadata:
  Creation Timestamp:  2021-12-11T04:41:59Z
  Generation:          2
  Managed Fields:
    API Version:  psmdb.percona.com/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:kubectl.kubernetes.io/last-applied-configuration:
      f:spec:
        .:
        f:psmdbCluster:
        f:storageName:
    Manager:      kubectl-client-side-apply
    Operation:    Update
    Time:         2021-12-11T04:41:59Z
    API Version:  psmdb.percona.com/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        .:
        f:error:
        f:state:
    Manager:         percona-server-mongodb-operator
    Operation:       Update
    Time:            2021-12-11T04:41:59Z
  Resource Version:  7628970
  UID:               ef4d522b-ccce-4838-867b-1344ad3d0437
Spec:
  Psmdb Cluster:  percona-mongo-psmdb-d
  Storage Name:   eu-west-1
Status:
  Error:  get cluster default/percona-mongo: PerconaServerMongoDB.psmdb.percona.com "percona-mongo" not found
  State:  error
Events:   <none>

Greatful for any hints how to troubleshoot further. (or a solution surfed on a silver platter sparkling in the moonlight :joy:)

1 Like
2

I did start from scratch, after found out the cluster was instable. Now installed everything in default namespace, both operator and cluster, and I don’t get above errors now anymore.

However, now if I want to login to Mongo, I am getting:

$ psmdb-db $ kubectl run -i --rm --tty percona-client ...

Error attaching, falling back to logs: unable to upgrade connection: container percona-client not found in pod percona-client_default

DNSHostNotFound: Failed to look up service "_mongodb._tcp.percona-db-psmdb-db-rs0.default.svc.cluster.local": Success

Altough this is not Backup related anymore I would be greatful for any hints. Description of psmdb returns ready states. I can login to the primary rs0-0 but not via the connection string referenced by percona helm charts without any replica index number.

    expose:
      enabled: true
      exposeType: ClusterIP

I switched expose.enabled: false and the rs0 service can be resolved now. Then I get:

{"t":{"$date":"2021-12-21T02:48:12.339Z"},"s":"I",  "c":"NETWORK",  "id":4333208, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"RSM host selection timeout","attr":{"replica
Set":"rs0","error":"FailedToSatisfyReadPreference: Could not find host matching read preference { mode: \"nearest\" } for set rs0"}}
Error: connect failed to replica set rs0/percona-db-psmdb-db-rs0-0.percona-db-psmdb-db-rs0.default.svc.cluster.local:27017,percona-db-psmdb-db-rs0-2.percona-db-psmdb-db-rs0
.default.svc.cluster.local:27017,percona-db-psmdb-db-rs0-1.percona-db-psmdb-db-rs0.default.svc.cluster.local:27017 :

I imagine I have to configure

replsets[0].configuration which is the custom config for mongod in replica set according to docs, how could one define ReadPreference in here to workaround the 4.4 bug I read about in a neighbouring thread. So I am looking at replication.localPingThresholdMs at https://docs.mongodb.com/manual/reference/configuration-options/ and Read Preference for Replica Sets https://docs.mongodb.com/manual/core/read-preference-mechanics/#std-label-replica-set-read-preference-behavior-member-selection because nearest option does not seem to work for me, even though primary should be default according to Percona docs.

One cannot help to wonder how these parameters set, would this happen to be correct parameters in values.yaml, i.e.:

replsets:
  - name: rs0
    size: 3
    configuration: 
       readPreference: primaryPreferred

Continuing this thread here: Could not find host matching read preference mode nearest for set rs0

1 Like