Check storage connecion: unable to get storage: get config: get: mongo: no documents in result

I’m using percona mongodb kubernetes operator; But I’m getting error while taking backup;
I’ve used manual backup i.e., by applying backup.yaml file available in operator docs;

When I check for

pbm status

I’m getting error as:

2021-05-17T08:18:30Z E [cfg/mongo-db-cfg-1.mongo-db-cfg.mongodb.svc.cluster.local:27017] [agentCheckup] check storage connecion: unable to get storage: get config: get: mongo: no documents in result

1 Like

Hi,

The error could be benign if it’s a new bucket. Could you check the backup status with kubectl get psmdb-backup and kubectl describe psmdb-backup <backupname>?

1 Like

When I describe it shows this in the error message

Error: couldn’t get response from all shards: convergeClusterWithTimeout: reached converge timeout

1 Like

I don’t think the cluster is in healthy state. Could you check the cluster status (kubectl describe psmdb) and operator logs?

1 Like

image

Operator logs

{“level”:“error”,“ts”:1621243884.3027275,“logger”:“controller_psmdb”,“msg”:“failed to reconcile cluster”,“Request.Namespace”:“mongodb”,“Request.Name”:“mongo-db”,“replset”:“cfg”,“error”:“dial:: failed to ping mongo: connection(mongo-db-cfg-1.mongo-db-cfg.mongodb.svc.cluster.local:27017[-73121]) incomplete read of message header: read tcp: i/o timeout”}

1 Like

Could you share the cr.yaml (and other modified files) so I can try to reproduce it?

1 Like

cr.yaml

apiVersion: psmdb.percona.com/v1-8-0
kind: PerconaServerMongoDB
metadata:
    labels:
        app: mongodb
    name: mongo-db
    namespace: mongodb
spec:
    crVersion: 1.8.0
    image: percona/percona-server-mongodb:4.4.5-7
    imagePullPolicy: Always
    allowUnsafeConfigurations: false
    updateStrategy: SmartUpdate
    upgradeOptions:
        versionServiceEndpoint: https://check.percona.com
        apply: 4.4-recommended
        schedule: 0 2 * * *
        setFCV: false
    secrets:
        users: mongo-db-secrets
    pmm:
        enabled: false
        image: percona/pmm-client:2.12.0
        serverHost: monitoring-service
    replsets:
        - name: rs0
          size: 3
          podDisruptionBudget:
              maxUnavailable: 1
          expose:
              enabled: false
              exposeType: LoadBalancer
          arbiter:
              enabled: false
              size: 1
          resources:
              limits:
                  cpu: 50m
                  memory: 400Mi
              requests:
                  cpu: 50m
                  memory: 400Mi
          volumeSpec:
              persistentVolumeClaim:
                  storageClassName: topolvm-provisioner
                  accessModes:
                      - ReadWriteOnce
                  resources:
                      requests:
                          storage: 2Gi
    sharding:
        enabled: true
        configsvrReplSet:
            size: 3
            podDisruptionBudget:
                maxUnavailable: 1
            resources:
                limits:
                    cpu: 50m
                    memory: 400Mi
                requests:
                    cpu: 50m
                    memory: 400Mi
            volumeSpec:
                persistentVolumeClaim:
                    storageClassName: topolvm-provisioner
                    accessModes:
                        - ReadWriteOnce
                    resources:
                        requests:
                            storage: 2Gi
        mongos:
            size: 3
            podDisruptionBudget:
                maxUnavailable: 1
            resources:
                limits:
                    cpu: 50m
                    memory: 400Mi
                requests:
                    cpu: 50m
                    memory: 400Mi
            expose:
                exposeType: ClusterIP
    mongod:
        net:
            port: 27017
            hostPort: 0
        security:
            redactClientLogData: false
            enableEncryption: true
            encryptionKeySecret: mongo-db-encryption-key
            encryptionCipherMode: AES256-CBC
        setParameter:
            ttlMonitorSleepSecs: 60
            wiredTigerConcurrentReadTransactions: 128
            wiredTigerConcurrentWriteTransactions: 128
        storage:
            engine: wiredTiger
            inMemory:
                engineConfig:
                    inMemorySizeRatio: 0.9
            wiredTiger:
                engineConfig:
                    cacheSizeRatio: 0.5
                    directoryForIndexes: false
                    journalCompressor: snappy
                collectionConfig:
                    blockCompressor: snappy
                indexConfig:
                    prefixCompression: true
            operationProfiling:
                mode: slowOp
                slowOpThresholdMs: 100
                rateLimit: 100
    backup:
        enabled: true
        restartOnFailure: true
        image: percona/percona-server-mongodb-operator:1.8.0-backup
        serviceAccountName: percona-server-mongodb-operator
        storages:
            mongo-daily-backup:
                type: s3
                s3:
                    bucket: mongodb-daily-bkp
                    region: us-east-1
                    credentialsSecret: mongodb-cluster-backup-secret
                    endpointUrl: https://<minio-svc>.<minio-ns>.svc.cluster.local:9000
            mongo-weekly-backup:
                type: s3
                s3:
                    bucket: mongodb-weekly-bkp
                    region: us-east-1
                    credentialsSecret: mongodb-cluster-backup-secret
                    endpointUrl: https://<minio-svc>.<minio-ns>.svc.cluster.local:9000
        pitr:
            enabled: false
        tasks:
            - name: daily-backup
              enabled: true
              schedule: 0 0 * * *
              keep: 3
              storageName: mongo-daily-backup
              compressionType: gzip
            - name: weekly-backup
              enabled: true
              schedule: 0 0 * * 6
              keep: 5
              storageName: mongo-weekly-backup
              compressionType: gzip

backup.yaml

apiVersion: psmdb.percona.com/v1
kind: PerconaServerMongoDBBackup
metadata:
  name: mongodb-test-backup
spec:
  psmdbCluster: mongo-db
  storageName: mongo-daily-backup

secrets.yaml

apiVersion: v1
kind: Secret
metadata:
    name: mongo-db-secrets
    namespace: mongodb
type: Opaque
stringData:
    MONGODB_BACKUP_USER: backup
    MONGODB_BACKUP_PASSWORD: backup123456
    MONGODB_CLUSTER_ADMIN_USER: clusterAdmin
    MONGODB_CLUSTER_ADMIN_PASSWORD: clusterAdmin123456
    MONGODB_CLUSTER_MONITOR_USER: clusterMonitor
    MONGODB_CLUSTER_MONITOR_PASSWORD: clusterMonitor123456
    MONGODB_USER_ADMIN_USER: userAdmin
    MONGODB_USER_ADMIN_PASSWORD: userAdmin123456
    PMM_SERVER_USER: admin
    PMM_SERVER_PASSWORD: admin

apiVersion: v1
kind: Secret
metadata:
    name: mongodb-cluster-backup-secret
    namespace: mongodb
type: Opaque
stringData:
    AWS_ACCESS_KEY_ID:<bucket_access_key>
    AWS_SECRET_ACCESS_KEY: <bucket_secret_key>
1 Like

@egegunes What should be the bucket policy for minio buckets created?

1 Like

Hello @SpoorthiPalakshaiah ,

I did the following - tried to configure two buckets in our operator - one Minio, one GCP.

GCP backup worked like a charm.
Minio is throwing an error.

I’m not an expert in Minio, but seems I’m doing something wrong. I tried their operator with defaults, created the bucket and using the default user and key.

But at least now we know that the problem is with Minio. We have an automated e2e test and it works just fine.
Do you have any issues connection with mc through S3 protocol to your minio server? Does it work and you can create the files?

1 Like

@spronin I can create files in the minio buckets using mc client; There is no issues in the connection with mc

1 Like

I have booked myself for tomorrow to test the minio and PSMDB Operator. It works, but I want to see the condition where it does not. Will keep you posted.

1 Like

Ok. Here are my findings:

  1. minio-operator by default is exposed with https only. I have not found the way to disable TLS.
    So in the configuration of PSMDB Operator you should have https link:

endpointUrl: https://minio.minio.svc.cluster.local:443

If you try to take the backup by creating the CR - the backup will fail with the following error:

 error: |-
    mongodump: write data: upload to S3: RequestError: send request failed
    caused by: Put "https://minio.minio.svc.cluster.local:443/sp-test/2021-05-25T11%3A31%3A25Z_cfg.dump.gz": x509: certificate signed by unknown authority.

The fix is obvious - apply the valid certificate. Minio Operator has the option.

One of the features that we can implement in our PSMDB Operator though is to disable cert check for PBM. I will create the ticket for it.

  1. If I install Minio from old helm chart (which is considered deprecated):
helm repo add minio https://helm.min.io/
helm install \
  --namespace minio \
  --generate-name \
  --set accessKey=mykey \
  --set secretKey=mysecretkey \
  --set service.type=ClusterIP \
  --set configPath=/tmp/.minio/ \
  --set persistence.size=2G \
  --set environment.MINIO_REGION=us-east-1 \
  minio/minio

I can reach minio via http on port 9000 and my endpointUrl would look like this:
endpointUrl: http://minio-1621944803.minio.svc.cluster.local:9000

In that case backups work fine with no additional tweaks:

$ ./mc ls myminio/sp-test
[2021-05-25 15:19:07 MSK]     5B .pbm.init
[2021-05-25 15:19:31 MSK] 2.0KiB 2021-05-25T12:19:06Z.pbm.json
[2021-05-25 15:19:26 MSK] 106KiB 2021-05-25T12:19:06Z_cfg.dump.gz
[2021-05-25 15:19:29 MSK]   998B 2021-05-25T12:19:06Z_cfg.oplog.gz
[2021-05-25 15:19:25 MSK]  12KiB 2021-05-25T12:19:06Z_rs0.dump.gz
[2021-05-25 15:19:31 MSK]   216B 2021-05-25T12:19:06Z_rs0.oplog.gz

Now I’m confused with your configuration, @SpoorthiPalakshaiah .

  1. How have you deployed minio? Please provide detailed steps and show the Service objects in k8s that you have for minio.
  2. Do you have it exposed via http or https?
  3. If you say that mc client can work - how do you do it? What is the exact command with full connection string and ports that you use to add an alias? Do you use --insecure flag?

As you see minio works fine, so it depends on your config and has nothing to do with PSMDB Operator.

1 Like
  1. I’ve deployed minio using Tenant; Service objects in k8s I’ve in minio

    kubectl get svc -n minio

NAME             TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)             AGE
console          ClusterIP   10.99.206.197    <none>        9090/TCP,9443/TCP   11d
minio            ClusterIP   10.98.90.91      <none>        443/TCP             11d
operator         ClusterIP   10.111.225.139   <none>        4222/TCP,4233/TCP   11d
minio-console   ClusterIP   10.97.53.216     <none>        9443/TCP            11d
minio-hl        ClusterIP   None             <none>        9000/TCP            11d
  1. I was trying it with minio-hl.minio.svc.cluster.local:9000
  2. I used mc command with public URL;

If this error pops up

error: |-
mongodump: write data: upload to S3: RequestError: send request failed
caused by: Put “https://minio.minio.svc.cluster.local:443/sp-test/2021-05-25T11%3A31%3A25Z_cfg.dump.gz”: x509: certificate signed by unknown authority.

I get this error when I use endpoint url as

https://minio-hl.minio.svc.cluster.local:9000

Where should be the certificates included;
Where should I set the certificates? I didn’t find any configs where we can include certificates

1 Like

This section in values.yaml of Minio helm chart allows you to configure the certs.

Example can be found here: minio/docs/tls/kubernetes at master · minio/minio · GitHub

1 Like

I have certs in minio; Where should I add them in mongodb?

1 Like

If these certs are not self-issued and valid - you don’t need to add them to mongodb.

1 Like

Is there a way to disable cert check?

1 Like

As I mentioned before - no.
Underlying tool (Percona Backup for MongoDB) does not support that yet. We are going to implement that in later releases.

There are two workarounds:

  1. Run Minio without https (not sure how to do that with Operator)
  2. Run Minio with valid public cert
1 Like

When I use minio with valid public cert:
.pbm.init file gets into the bucket but the backup shows error as

couldn’t get response from all shards: convergeClusterWithTimeout: reached converge timeout

1 Like

@spronin Hi, we are running minio with valid public cert; My question is should we need to include that crt and key anywhere in mongodb configurations?

1 Like