# Can't do backups in K8s cluster

**URL:** <https://forums.percona.com/t/cant-do-backups-in-k8s-cluster/13499>\
**Category:** Percona Backup for MongoDB\
**Created:** [December 15, 2021, 8:41pm UTC](https://forums.percona.com/t/cant-do-backups-in-k8s-cluster/13499 "2021-12-15T20:41:13Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![anon17965173](https://avatars.discourse-cdn.com/v4/letter/a/278dde/32.png) [@anon17965173](https://forums.percona.com/u/anon17965173)\
**Post date:** [December 15, 2021, 8:41pm UTC](https://forums.percona.com/t/cant-do-backups-in-k8s-cluster/13499/1 "2021-12-15T20:41:13Z")

</div>

I’m trying to establish a PITR to Minio. Both replicated, no sharding.

These are the logs:

**psmdb-db**

```auto
    Last Transition Time: 2021-12-15T13:37:24Z
    Status: True
    Type: ready
    Last Transition Time: 2021-12-15T13:38:46Z
    Message: create pbm object: create PBM connection to percona-mongo-psmdb-d-rs0-2.percona-mongo-psmdb-d-rs0.default.svc.cluster.local:27017,percona-mongo-psmdb-d-rs0-0.percona-mongo-psmdb-d-rs0.default.svc.cluster.local:27017
: create mongo connection: mongo ping: server selection error: server selection timeout, current topology: { Type: Unknown, Servers: [{ Addr: percona-mongo-psmdb-d-rs0-2.percona-mongo-psmdb-d-rs0.default.svc.cluster.local:27017, Type: Unknown
, Last error: connection() error occured during connection handshake: dial tcp: lookup percona-mongo-psmdb-d-rs0-2.percona-mongo-psmdb-d-rs0.default.svc.cluster.local on 169.254.20.10:53: no such host }, { Addr: percona-mongo-psmdb-d-rs0-0.pe
rcona-mongo-psmdb-d-rs0.default.svc.cluster.local:27017, Type: RSGhost, Average RTT: 2332184 }, ] }
    Reason: ErrorReconcile
    Status: True
    Type: error
    Last Transition Time: 2021-12-15T13:38:47Z
    Status: True
    Type: initializing
    Last Transition Time: 2021-12-15T13:39:37Z
    Message: rs0: ready
    Reason: RSReady
    Status: True
    Type: ready
    Last Transition Time: 2021-12-15T13:39:37Z
    Status: True
    Type: initializing
    Last Transition Time: 2021-12-15T13:39:42Z
    Status: True
    Type: ready
  Host: percona-mongo-psmdb-d-rs0.default.svc.cluster.local
  Mongo Image: percona/percona-server-mongodb:4.4.8-9
  Mongo Version: 4.4.8-9
  Observed Generation: 11
  Pmm Version: 2.21.0
  Ready: 3
  Replsets:
    rs0:
      Initialized: true
      Ready: 3
      Size: 3
      Status: ready
  Size: 3
  State: ready
Events: <none>
psmdb-db $

```

**spec.backup.storages**

```auto
  minio:
      type: s3
      s3:
        bucket: bucket-1
        credentialsSecret: myskills-minio-bucket
        endpointUrl: https://minio.default.svc.cluster.local:443

# fs-pvc:
# type: filesystem
# volume:
# persistentVolumeClaim:
# storageClassName: longhorn-static
# accessModes: ["ReadWriteOnce"]
# resources:
# requests:
# storage: 10Gi

  pitr:
    enabled: true
  tasks:
   - name: daily-psmdb-backup
     enabled: true
     schedule: "0 0 * * *"
     keep: 3
     storageName: minio
     compressionType: gzip

```

**psmdb-backup**

```auto
Name: backup1
Namespace: default
Labels: <none>
Annotations: <none>
API Version: psmdb.percona.com/v1
Kind: PerconaServerMongoDBBackup
Metadata:
  Creation Timestamp: 2021-12-11T04:41:59Z
  Generation: 2
  Managed Fields:
    API Version: psmdb.percona.com/v1
    Fields Type: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:kubectl.kubernetes.io/last-applied-configuration:
      f:spec:
        .:
        f:psmdbCluster:
        f:storageName:
    Manager: kubectl-client-side-apply
    Operation: Update
    Time: 2021-12-11T04:41:59Z
    API Version: psmdb.percona.com/v1
    Fields Type: FieldsV1
    fieldsV1:
      f:status:
        .:
        f:error:
        f:state:
    Manager: percona-server-mongodb-operator
    Operation: Update
    Time: 2021-12-11T04:41:59Z
  Resource Version: 7628970
  UID: ef4d522b-ccce-4838-867b-1344ad3d0437
Spec:
  Psmdb Cluster: percona-mongo-psmdb-d
  Storage Name: eu-west-1
Status:
  Error: get cluster default/percona-mongo: PerconaServerMongoDB.psmdb.percona.com "percona-mongo" not found
  State: error
Events: <none>

```

Greatful for any hints how to troubleshoot further. (or a solution surfed on a silver platter sparkling in the moonlight 😂)

---

<div class="post-metadata">

**Author:** ![anon17965173](https://avatars.discourse-cdn.com/v4/letter/a/278dde/32.png) [@anon17965173](https://forums.percona.com/u/anon17965173)\
**Post date:** [December 21, 2021, 2:16am UTC](https://forums.percona.com/t/cant-do-backups-in-k8s-cluster/13499/2 "2021-12-21T02:16:03Z")

</div>

I did start from scratch, after found out the cluster was instable. Now installed everything in default namespace, both operator and cluster, and I don’t get above errors now anymore.

However, now if I want to login to Mongo, I am getting:

`$ psmdb-db $ kubectl run -i --rm --tty percona-client ...`

```auto
Error attaching, falling back to logs: unable to upgrade connection: container percona-client not found in pod percona-client_default

DNSHostNotFound: Failed to look up service "_mongodb._tcp.percona-db-psmdb-db-rs0.default.svc.cluster.local": Success

```

Altough this is not Backup related anymore I would be greatful for any hints. Description of psmdb returns ready states. I can login to the primary rs0-0 but not via the connection string referenced by percona helm charts without any replica index number.

```auto
    expose:
      enabled: true
      exposeType: ClusterIP

```

I switched `expose.enabled: false` and the `rs0` service can be resolved now. Then I get:

```auto
{"t":{"$date":"2021-12-21T02:48:12.339Z"},"s":"I", "c":"NETWORK", "id":4333208, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"RSM host selection timeout","attr":{"replica
Set":"rs0","error":"FailedToSatisfyReadPreference: Could not find host matching read preference { mode: \"nearest\" } for set rs0"}}
Error: connect failed to replica set rs0/percona-db-psmdb-db-rs0-0.percona-db-psmdb-db-rs0.default.svc.cluster.local:27017,percona-db-psmdb-db-rs0-2.percona-db-psmdb-db-rs0
.default.svc.cluster.local:27017,percona-db-psmdb-db-rs0-1.percona-db-psmdb-db-rs0.default.svc.cluster.local:27017 :

```

I imagine I have to configure

`replsets[0].configuration` which is the `custom config for mongod in replica set ` according to docs, how could one define `ReadPreference` in here to workaround the 4.4 bug I read about in a neighbouring thread. So I am looking at `replication.localPingThresholdMs` at [https://docs.mongodb.com/manual/reference/configuration-options/](https://docs.mongodb.com/manual/reference/configuration-options/) and **Read Preference for Replica Sets** [https://docs.mongodb.com/manual/core/read-preference-mechanics/#std-label-replica-set-read-preference-behavior-member-selection](https://docs.mongodb.com/manual/core/read-preference-mechanics/#std-label-replica-set-read-preference-behavior-member-selection) because `nearest` option does not seem to work for me, even though primary should be default according to Percona docs.

One cannot help to wonder how these parameters set, would this happen to be correct parameters in values.yaml, i.e.:

```auto
replsets:
  - name: rs0
    size: 3
    configuration: 
       readPreference: primaryPreferred

```

Continuing this thread here: [Could not find host matching read preference mode nearest for set rs0](https://forums.percona.com/t/could-not-find-host-matching-read-preference-mode-nearest-for-set-rs0/13596)