Check storage connecion: unable to get storage: get config: get: mongo: no documents in result

MongoDB Percona Operator for MongoDB
1

I’m using percona mongodb kubernetes operator; But I’m getting error while taking backup;
I’ve used manual backup i.e., by applying backup.yaml file available in operator docs;

When I check for

pbm status

I’m getting error as:

2021-05-17T08:18:30Z E [cfg/mongo-db-cfg-1.mongo-db-cfg.mongodb.svc.cluster.local:27017] [agentCheckup] check storage connecion: unable to get storage: get config: get: mongo: no documents in result

1 Like
2

Hi,

The error could be benign if it’s a new bucket. Could you check the backup status with kubectl get psmdb-backup and kubectl describe psmdb-backup <backupname>?

1 Like
3

When I describe it shows this in the error message

Error: couldn’t get response from all shards: convergeClusterWithTimeout: reached converge timeout

1 Like
4

I don’t think the cluster is in healthy state. Could you check the cluster status (kubectl describe psmdb) and operator logs?

1 Like
5

image

Operator logs

{“level”:“error”,“ts”:1621243884.3027275,“logger”:“controller_psmdb”,“msg”:“failed to reconcile cluster”,“Request.Namespace”:“mongodb”,“Request.Name”:“mongo-db”,“replset”:“cfg”,“error”:“dial:: failed to ping mongo: connection(mongo-db-cfg-1.mongo-db-cfg.mongodb.svc.cluster.local:27017[-73121]) incomplete read of message header: read tcp: i/o timeout”}

1 Like
6

Could you share the cr.yaml (and other modified files) so I can try to reproduce it?

1 Like
7

cr.yaml

apiVersion: psmdb.percona.com/v1-8-0
kind: PerconaServerMongoDB
metadata:
    labels:
        app: mongodb
    name: mongo-db
    namespace: mongodb
spec:
    crVersion: 1.8.0
    image: percona/percona-server-mongodb:4.4.5-7
    imagePullPolicy: Always
    allowUnsafeConfigurations: false
    updateStrategy: SmartUpdate
    upgradeOptions:
        versionServiceEndpoint: https://check.percona.com
        apply: 4.4-recommended
        schedule: 0 2 * * *
        setFCV: false
    secrets:
        users: mongo-db-secrets
    pmm:
        enabled: false
        image: percona/pmm-client:2.12.0
        serverHost: monitoring-service
    replsets:
        - name: rs0
          size: 3
          podDisruptionBudget:
              maxUnavailable: 1
          expose:
              enabled: false
              exposeType: LoadBalancer
          arbiter:
              enabled: false
              size: 1
          resources:
              limits:
                  cpu: 50m
                  memory: 400Mi
              requests:
                  cpu: 50m
                  memory: 400Mi
          volumeSpec:
              persistentVolumeClaim:
                  storageClassName: topolvm-provisioner
                  accessModes:
                      - ReadWriteOnce
                  resources:
                      requests:
                          storage: 2Gi
    sharding:
        enabled: true
        configsvrReplSet:
            size: 3
            podDisruptionBudget:
                maxUnavailable: 1
            resources:
                limits:
                    cpu: 50m
                    memory: 400Mi
                requests:
                    cpu: 50m
                    memory: 400Mi
            volumeSpec:
                persistentVolumeClaim:
                    storageClassName: topolvm-provisioner
                    accessModes:
                        - ReadWriteOnce
                    resources:
                        requests:
                            storage: 2Gi
        mongos:
            size: 3
            podDisruptionBudget:
                maxUnavailable: 1
            resources:
                limits:
                    cpu: 50m
                    memory: 400Mi
                requests:
                    cpu: 50m
                    memory: 400Mi
            expose:
                exposeType: ClusterIP
    mongod:
        net:
            port: 27017
            hostPort: 0
        security:
            redactClientLogData: false
            enableEncryption: true
            encryptionKeySecret: mongo-db-encryption-key
            encryptionCipherMode: AES256-CBC
        setParameter:
            ttlMonitorSleepSecs: 60
            wiredTigerConcurrentReadTransactions: 128
            wiredTigerConcurrentWriteTransactions: 128
        storage:
            engine: wiredTiger
            inMemory:
                engineConfig:
                    inMemorySizeRatio: 0.9
            wiredTiger:
                engineConfig:
                    cacheSizeRatio: 0.5
                    directoryForIndexes: false
                    journalCompressor: snappy
                collectionConfig:
                    blockCompressor: snappy
                indexConfig:
                    prefixCompression: true
            operationProfiling:
                mode: slowOp
                slowOpThresholdMs: 100
                rateLimit: 100
    backup:
        enabled: true
        restartOnFailure: true
        image: percona/percona-server-mongodb-operator:1.8.0-backup
        serviceAccountName: percona-server-mongodb-operator
        storages:
            mongo-daily-backup:
                type: s3
                s3:
                    bucket: mongodb-daily-bkp
                    region: us-east-1
                    credentialsSecret: mongodb-cluster-backup-secret
                    endpointUrl: https://<minio-svc>.<minio-ns>.svc.cluster.local:9000
            mongo-weekly-backup:
                type: s3
                s3:
                    bucket: mongodb-weekly-bkp
                    region: us-east-1
                    credentialsSecret: mongodb-cluster-backup-secret
                    endpointUrl: https://<minio-svc>.<minio-ns>.svc.cluster.local:9000
        pitr:
            enabled: false
        tasks:
            - name: daily-backup
              enabled: true
              schedule: 0 0 * * *
              keep: 3
              storageName: mongo-daily-backup
              compressionType: gzip
            - name: weekly-backup
              enabled: true
              schedule: 0 0 * * 6
              keep: 5
              storageName: mongo-weekly-backup
              compressionType: gzip

backup.yaml

apiVersion: psmdb.percona.com/v1
kind: PerconaServerMongoDBBackup
metadata:
  name: mongodb-test-backup
spec:
  psmdbCluster: mongo-db
  storageName: mongo-daily-backup

secrets.yaml

apiVersion: v1
kind: Secret
metadata:
    name: mongo-db-secrets
    namespace: mongodb
type: Opaque
stringData:
    MONGODB_BACKUP_USER: backup
    MONGODB_BACKUP_PASSWORD: backup123456
    MONGODB_CLUSTER_ADMIN_USER: clusterAdmin
    MONGODB_CLUSTER_ADMIN_PASSWORD: clusterAdmin123456
    MONGODB_CLUSTER_MONITOR_USER: clusterMonitor
    MONGODB_CLUSTER_MONITOR_PASSWORD: clusterMonitor123456
    MONGODB_USER_ADMIN_USER: userAdmin
    MONGODB_USER_ADMIN_PASSWORD: userAdmin123456
    PMM_SERVER_USER: admin
    PMM_SERVER_PASSWORD: admin

apiVersion: v1
kind: Secret
metadata:
    name: mongodb-cluster-backup-secret
    namespace: mongodb
type: Opaque
stringData:
    AWS_ACCESS_KEY_ID:<bucket_access_key>
    AWS_SECRET_ACCESS_KEY: <bucket_secret_key>
1 Like
8

@Ege_Gunes What should be the bucket policy for minio buckets created?

1 Like
9

Hello @SpoorthiPalakshaiah ,

I did the following - tried to configure two buckets in our operator - one Minio, one GCP.

GCP backup worked like a charm.
Minio is throwing an error.

I’m not an expert in Minio, but seems I’m doing something wrong. I tried their operator with defaults, created the bucket and using the default user and key.

But at least now we know that the problem is with Minio. We have an automated e2e test and it works just fine.
Do you have any issues connection with mc through S3 protocol to your minio server? Does it work and you can create the files?

1 Like
10

@Sergey_Pronin I can create files in the minio buckets using mc client; There is no issues in the connection with mc

1 Like
11

I have booked myself for tomorrow to test the minio and PSMDB Operator. It works, but I want to see the condition where it does not. Will keep you posted.

1 Like
12

Ok. Here are my findings:

  1. minio-operator by default is exposed with https only. I have not found the way to disable TLS.
    So in the configuration of PSMDB Operator you should have https link:

endpointUrl: https://minio.minio.svc.cluster.local:443

If you try to take the backup by creating the CR - the backup will fail with the following error:

 error: |-
    mongodump: write data: upload to S3: RequestError: send request failed
    caused by: Put "https://minio.minio.svc.cluster.local:443/sp-test/2021-05-25T11%3A31%3A25Z_cfg.dump.gz": x509: certificate signed by unknown authority.

The fix is obvious - apply the valid certificate. Minio Operator has the option.

One of the features that we can implement in our PSMDB Operator though is to disable cert check for PBM. I will create the ticket for it.

  1. If I install Minio from old helm chart (which is considered deprecated):
helm repo add minio https://helm.min.io/
helm install \
  --namespace minio \
  --generate-name \
  --set accessKey=mykey \
  --set secretKey=mysecretkey \
  --set service.type=ClusterIP \
  --set configPath=/tmp/.minio/ \
  --set persistence.size=2G \
  --set environment.MINIO_REGION=us-east-1 \
  minio/minio

I can reach minio via http on port 9000 and my endpointUrl would look like this:
endpointUrl: http://minio-1621944803.minio.svc.cluster.local:9000

In that case backups work fine with no additional tweaks:

$ ./mc ls myminio/sp-test
[2021-05-25 15:19:07 MSK]     5B .pbm.init
[2021-05-25 15:19:31 MSK] 2.0KiB 2021-05-25T12:19:06Z.pbm.json
[2021-05-25 15:19:26 MSK] 106KiB 2021-05-25T12:19:06Z_cfg.dump.gz
[2021-05-25 15:19:29 MSK]   998B 2021-05-25T12:19:06Z_cfg.oplog.gz
[2021-05-25 15:19:25 MSK]  12KiB 2021-05-25T12:19:06Z_rs0.dump.gz
[2021-05-25 15:19:31 MSK]   216B 2021-05-25T12:19:06Z_rs0.oplog.gz

Now I’m confused with your configuration, @SpoorthiPalakshaiah .

  1. How have you deployed minio? Please provide detailed steps and show the Service objects in k8s that you have for minio.
  2. Do you have it exposed via http or https?
  3. If you say that mc client can work - how do you do it? What is the exact command with full connection string and ports that you use to add an alias? Do you use --insecure flag?

As you see minio works fine, so it depends on your config and has nothing to do with PSMDB Operator.

1 Like
13
  1. I’ve deployed minio using Tenant; Service objects in k8s I’ve in minio

    kubectl get svc -n minio

NAME             TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)             AGE
console          ClusterIP   10.99.206.197    <none>        9090/TCP,9443/TCP   11d
minio            ClusterIP   10.98.90.91      <none>        443/TCP             11d
operator         ClusterIP   10.111.225.139   <none>        4222/TCP,4233/TCP   11d
minio-console   ClusterIP   10.97.53.216     <none>        9443/TCP            11d
minio-hl        ClusterIP   None             <none>        9000/TCP            11d
  1. I was trying it with minio-hl.minio.svc.cluster.local:9000
  2. I used mc command with public URL;

If this error pops up

error: |-
mongodump: write data: upload to S3: RequestError: send request failed
caused by: Put “https://minio.minio.svc.cluster.local:443/sp-test/2021-05-25T11%3A31%3A25Z_cfg.dump.gz”: x509: certificate signed by unknown authority.

I get this error when I use endpoint url as

https://minio-hl.minio.svc.cluster.local:9000

Where should be the certificates included;
Where should I set the certificates? I didn’t find any configs where we can include certificates

1 Like
14

This section in values.yaml of Minio helm chart allows you to configure the certs.

Example can be found here: minio/docs/tls/kubernetes at master · minio/minio · GitHub

1 Like
15

I have certs in minio; Where should I add them in mongodb?

1 Like
16

If these certs are not self-issued and valid - you don’t need to add them to mongodb.

1 Like
17

Is there a way to disable cert check?

1 Like
18

As I mentioned before - no.
Underlying tool (Percona Backup for MongoDB) does not support that yet. We are going to implement that in later releases.

There are two workarounds:

  1. Run Minio without https (not sure how to do that with Operator)
  2. Run Minio with valid public cert
1 Like
19

When I use minio with valid public cert:
.pbm.init file gets into the bucket but the backup shows error as

couldn’t get response from all shards: convergeClusterWithTimeout: reached converge timeout

1 Like
20

@Sergey_Pronin Hi, we are running minio with valid public cert; My question is should we need to include that crt and key anywhere in mongodb configurations?

1 Like