# Check storage connecion: unable to get storage: get config: get: mongo: no documents in result **URL:** \ **Category:** Percona Operator for MongoDB\ **Created:** [May 17, 2021, 8:32am UTC](https://forums.percona.com/t/check-storage-connecion-unable-to-get-storage-get-config-get-mongo-no-documents-in-result/10482 "2021-05-17T08:32:59Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![SpoorthiPalakshaiah](https://avatars.discourse-cdn.com/v4/letter/s/8dc957/32.png) [@SpoorthiPalakshaiah](https://forums.percona.com/u/SpoorthiPalakshaiah)\ **Post date:** [May 17, 2021, 8:32am UTC](https://forums.percona.com/t/check-storage-connecion-unable-to-get-storage-get-config-get-mongo-no-documents-in-result/10482/1 "2021-05-17T08:32:59Z")
I’m using percona mongodb kubernetes operator; But I’m getting error while taking backup; I’ve used manual backup i.e., by applying backup.yaml file available in operator docs; When I check for > pbm status I’m getting error as: > 2021-05-17T08:18:30Z E [cfg/mongo-db-cfg-1.mongo-db-cfg.mongodb.svc.cluster.local:27017] [agentCheckup] check storage connecion: unable to get storage: get config: get: mongo: no documents in result ---
**Author:** ![Ege\_Gunes](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/ege_gunes/32/4595_2.png) [@Ege\_Gunes](https://forums.percona.com/u/Ege_Gunes)\ **Post date:** [May 17, 2021, 8:40am UTC](https://forums.percona.com/t/check-storage-connecion-unable-to-get-storage-get-config-get-mongo-no-documents-in-result/10482/2 "2021-05-17T08:40:59Z")
Hi, The error could be benign if it’s a new bucket. Could you check the backup status with `kubectl get psmdb-backup` and `kubectl describe psmdb-backup `? ---
**Author:** ![SpoorthiPalakshaiah](https://avatars.discourse-cdn.com/v4/letter/s/8dc957/32.png) [@SpoorthiPalakshaiah](https://forums.percona.com/u/SpoorthiPalakshaiah)\ **Post date:** [May 17, 2021, 8:53am UTC](https://forums.percona.com/t/check-storage-connecion-unable-to-get-storage-get-config-get-mongo-no-documents-in-result/10482/3 "2021-05-17T08:53:09Z")
When I describe it shows this in the error message > Error: couldn’t get response from all shards: convergeClusterWithTimeout: reached converge timeout ---
**Author:** ![Ege\_Gunes](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/ege_gunes/32/4595_2.png) [@Ege\_Gunes](https://forums.percona.com/u/Ege_Gunes)\ **Post date:** [May 17, 2021, 9:00am UTC](https://forums.percona.com/t/check-storage-connecion-unable-to-get-storage-get-config-get-mongo-no-documents-in-result/10482/4 "2021-05-17T09:00:49Z")
I don’t think the cluster is in healthy state. Could you check the cluster status (`kubectl describe psmdb`) and operator logs? ---
**Author:** ![SpoorthiPalakshaiah](https://avatars.discourse-cdn.com/v4/letter/s/8dc957/32.png) [@SpoorthiPalakshaiah](https://forums.percona.com/u/SpoorthiPalakshaiah)\ **Post date:** [May 17, 2021, 9:44am UTC](https://forums.percona.com/t/check-storage-connecion-unable-to-get-storage-get-config-get-mongo-no-documents-in-result/10482/5 "2021-05-17T09:44:44Z")
> [@Ege\_Gunes](#): > > `kubectl describe psmdb` ) ![image](https://us1.discourse-cdn.com/flex019/uploads/percona1/original/2X/7/79dda0c2475d24e2cfb39aa849ad64c63ee75e22.png) Operator logs > {“level”:“error”,“ts”:1621243884.3027275,“logger”:“controller\_psmdb”,“msg”:“failed to reconcile cluster”,“Request.Namespace”:“mongodb”,“Request.Name”:“mongo-db”,“replset”:“cfg”,“error”:“dial:: failed to ping mongo: connection(mongo-db-cfg-1.mongo-db-cfg.mongodb.svc.cluster.local:27017[-73121]) incomplete read of message header: read tcp: i/o timeout”} ---
**Author:** ![Ege\_Gunes](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/ege_gunes/32/4595_2.png) [@Ege\_Gunes](https://forums.percona.com/u/Ege_Gunes)\ **Post date:** [May 17, 2021, 12:23pm UTC](https://forums.percona.com/t/check-storage-connecion-unable-to-get-storage-get-config-get-mongo-no-documents-in-result/10482/6 "2021-05-17T12:23:37Z")
Could you share the cr.yaml (and other modified files) so I can try to reproduce it? ---
**Author:** ![SpoorthiPalakshaiah](https://avatars.discourse-cdn.com/v4/letter/s/8dc957/32.png) [@SpoorthiPalakshaiah](https://forums.percona.com/u/SpoorthiPalakshaiah)\ **Post date:** [May 17, 2021, 12:39pm UTC](https://forums.percona.com/t/check-storage-connecion-unable-to-get-storage-get-config-get-mongo-no-documents-in-result/10482/7 "2021-05-17T12:39:25Z")
cr.yaml ``` apiVersion: psmdb.percona.com/v1-8-0 kind: PerconaServerMongoDB metadata: labels: app: mongodb name: mongo-db namespace: mongodb spec: crVersion: 1.8.0 image: percona/percona-server-mongodb:4.4.5-7 imagePullPolicy: Always allowUnsafeConfigurations: false updateStrategy: SmartUpdate upgradeOptions: versionServiceEndpoint: https://check.percona.com apply: 4.4-recommended schedule: 0 2 * * * setFCV: false secrets: users: mongo-db-secrets pmm: enabled: false image: percona/pmm-client:2.12.0 serverHost: monitoring-service replsets: - name: rs0 size: 3 podDisruptionBudget: maxUnavailable: 1 expose: enabled: false exposeType: LoadBalancer arbiter: enabled: false size: 1 resources: limits: cpu: 50m memory: 400Mi requests: cpu: 50m memory: 400Mi volumeSpec: persistentVolumeClaim: storageClassName: topolvm-provisioner accessModes: - ReadWriteOnce resources: requests: storage: 2Gi sharding: enabled: true configsvrReplSet: size: 3 podDisruptionBudget: maxUnavailable: 1 resources: limits: cpu: 50m memory: 400Mi requests: cpu: 50m memory: 400Mi volumeSpec: persistentVolumeClaim: storageClassName: topolvm-provisioner accessModes: - ReadWriteOnce resources: requests: storage: 2Gi mongos: size: 3 podDisruptionBudget: maxUnavailable: 1 resources: limits: cpu: 50m memory: 400Mi requests: cpu: 50m memory: 400Mi expose: exposeType: ClusterIP mongod: net: port: 27017 hostPort: 0 security: redactClientLogData: false enableEncryption: true encryptionKeySecret: mongo-db-encryption-key encryptionCipherMode: AES256-CBC setParameter: ttlMonitorSleepSecs: 60 wiredTigerConcurrentReadTransactions: 128 wiredTigerConcurrentWriteTransactions: 128 storage: engine: wiredTiger inMemory: engineConfig: inMemorySizeRatio: 0.9 wiredTiger: engineConfig: cacheSizeRatio: 0.5 directoryForIndexes: false journalCompressor: snappy collectionConfig: blockCompressor: snappy indexConfig: prefixCompression: true operationProfiling: mode: slowOp slowOpThresholdMs: 100 rateLimit: 100 backup: enabled: true restartOnFailure: true image: percona/percona-server-mongodb-operator:1.8.0-backup serviceAccountName: percona-server-mongodb-operator storages: mongo-daily-backup: type: s3 s3: bucket: mongodb-daily-bkp region: us-east-1 credentialsSecret: mongodb-cluster-backup-secret endpointUrl: https://..svc.cluster.local:9000 mongo-weekly-backup: type: s3 s3: bucket: mongodb-weekly-bkp region: us-east-1 credentialsSecret: mongodb-cluster-backup-secret endpointUrl: https://..svc.cluster.local:9000 pitr: enabled: false tasks: - name: daily-backup enabled: true schedule: 0 0 * * * keep: 3 storageName: mongo-daily-backup compressionType: gzip - name: weekly-backup enabled: true schedule: 0 0 * * 6 keep: 5 storageName: mongo-weekly-backup compressionType: gzip ``` backup.yaml ``` apiVersion: psmdb.percona.com/v1 kind: PerconaServerMongoDBBackup metadata: name: mongodb-test-backup spec: psmdbCluster: mongo-db storageName: mongo-daily-backup ``` secrets.yaml ``` apiVersion: v1 kind: Secret metadata: name: mongo-db-secrets namespace: mongodb type: Opaque stringData: MONGODB_BACKUP_USER: backup MONGODB_BACKUP_PASSWORD: backup123456 MONGODB_CLUSTER_ADMIN_USER: clusterAdmin MONGODB_CLUSTER_ADMIN_PASSWORD: clusterAdmin123456 MONGODB_CLUSTER_MONITOR_USER: clusterMonitor MONGODB_CLUSTER_MONITOR_PASSWORD: clusterMonitor123456 MONGODB_USER_ADMIN_USER: userAdmin MONGODB_USER_ADMIN_PASSWORD: userAdmin123456 PMM_SERVER_USER: admin PMM_SERVER_PASSWORD: admin ``` * * * ``` apiVersion: v1 kind: Secret metadata: name: mongodb-cluster-backup-secret namespace: mongodb type: Opaque stringData: AWS_ACCESS_KEY_ID: AWS_SECRET_ACCESS_KEY: ``` ---
**Author:** ![SpoorthiPalakshaiah](https://avatars.discourse-cdn.com/v4/letter/s/8dc957/32.png) [@SpoorthiPalakshaiah](https://forums.percona.com/u/SpoorthiPalakshaiah)\ **Post date:** [May 17, 2021, 2:53pm UTC](https://forums.percona.com/t/check-storage-connecion-unable-to-get-storage-get-config-get-mongo-no-documents-in-result/10482/8 "2021-05-17T14:53:49Z")
@Ege_Gunes What should be the bucket policy for minio buckets created? ---
**Author:** ![Sergey\_Pronin](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/sergey_pronin/32/14887_2.png) [@Sergey\_Pronin](https://forums.percona.com/u/Sergey_Pronin)\ **Post date:** [May 18, 2021, 8:57pm UTC](https://forums.percona.com/t/check-storage-connecion-unable-to-get-storage-get-config-get-mongo-no-documents-in-result/10482/9 "2021-05-18T20:57:32Z")
Hello @SpoorthiPalakshaiah , I did the following - tried to configure two buckets in our operator - one Minio, one GCP. GCP backup worked like a charm. Minio is throwing an error. I’m not an expert in Minio, but seems I’m doing something wrong. I tried their operator with defaults, created the bucket and using the default user and key. But at least now we know that the problem is with Minio. We have an automated e2e test and it works just fine. Do you have any issues connection with mc through S3 protocol to your minio server? Does it work and you can create the files? ---
**Author:** ![SpoorthiPalakshaiah](https://avatars.discourse-cdn.com/v4/letter/s/8dc957/32.png) [@SpoorthiPalakshaiah](https://forums.percona.com/u/SpoorthiPalakshaiah)\ **Post date:** [May 19, 2021, 6:03am UTC](https://forums.percona.com/t/check-storage-connecion-unable-to-get-storage-get-config-get-mongo-no-documents-in-result/10482/10 "2021-05-19T06:03:23Z")
@Sergey_Pronin I can create files in the minio buckets using mc client; There is no issues in the connection with mc ---
**Author:** ![Sergey\_Pronin](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/sergey_pronin/32/14887_2.png) [@Sergey\_Pronin](https://forums.percona.com/u/Sergey_Pronin)\ **Post date:** [May 24, 2021, 11:27am UTC](https://forums.percona.com/t/check-storage-connecion-unable-to-get-storage-get-config-get-mongo-no-documents-in-result/10482/11 "2021-05-24T11:27:41Z")
I have booked myself for tomorrow to test the minio and PSMDB Operator. It works, but I want to see the condition where it does not. Will keep you posted. ---
**Author:** ![Sergey\_Pronin](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/sergey_pronin/32/14887_2.png) [@Sergey\_Pronin](https://forums.percona.com/u/Sergey_Pronin)\ **Post date:** [May 25, 2021, 12:31pm UTC](https://forums.percona.com/t/check-storage-connecion-unable-to-get-storage-get-config-get-mongo-no-documents-in-result/10482/12 "2021-05-25T12:31:12Z")
Ok. Here are my findings: 1. [minio-operator](https://github.com/minio/operator) by default is exposed with https only. I have not found the way to disable TLS. So in the configuration of PSMDB Operator you should have https link: `endpointUrl: https://minio.minio.svc.cluster.local:443` If you try to take the backup by creating the CR - the backup will fail with the following error: ```auto error: |- mongodump: write data: upload to S3: RequestError: send request failed caused by: Put "https://minio.minio.svc.cluster.local:443/sp-test/2021-05-25T11%3A31%3A25Z_cfg.dump.gz": x509: certificate signed by unknown authority. ``` The fix is obvious - apply the valid certificate. Minio Operator has the option. One of the features that we can implement in our PSMDB Operator though is to disable cert check for PBM. I will create the ticket for it. 1. If I install Minio from old helm chart (which is considered deprecated): ```auto helm repo add minio https://helm.min.io/ helm install \ --namespace minio \ --generate-name \ --set accessKey=mykey \ --set secretKey=mysecretkey \ --set service.type=ClusterIP \ --set configPath=/tmp/.minio/ \ --set persistence.size=2G \ --set environment.MINIO_REGION=us-east-1 \ minio/minio ``` I can reach minio via http on port 9000 and my endpointUrl would look like this: `endpointUrl: http://minio-1621944803.minio.svc.cluster.local:9000` In that case backups work fine with no additional tweaks: ```auto $ ./mc ls myminio/sp-test [2021-05-25 15:19:07 MSK] 5B .pbm.init [2021-05-25 15:19:31 MSK] 2.0KiB 2021-05-25T12:19:06Z.pbm.json [2021-05-25 15:19:26 MSK] 106KiB 2021-05-25T12:19:06Z_cfg.dump.gz [2021-05-25 15:19:29 MSK] 998B 2021-05-25T12:19:06Z_cfg.oplog.gz [2021-05-25 15:19:25 MSK] 12KiB 2021-05-25T12:19:06Z_rs0.dump.gz [2021-05-25 15:19:31 MSK] 216B 2021-05-25T12:19:06Z_rs0.oplog.gz ``` Now I’m confused with your configuration, @SpoorthiPalakshaiah . 1. How have you deployed minio? Please provide detailed steps and show the Service objects in k8s that you have for minio. 2. Do you have it exposed via http or https? 3. If you say that mc client can work - how do you do it? What is the exact command with full connection string and ports that you use to add an alias? Do you use --insecure flag? As you see minio works fine, so it depends on your config and has nothing to do with PSMDB Operator. ---
**Author:** ![SpoorthiPalakshaiah](https://avatars.discourse-cdn.com/v4/letter/s/8dc957/32.png) [@SpoorthiPalakshaiah](https://forums.percona.com/u/SpoorthiPalakshaiah)\ **Post date:** [May 25, 2021, 2:28pm UTC](https://forums.percona.com/t/check-storage-connecion-unable-to-get-storage-get-config-get-mongo-no-documents-in-result/10482/13 "2021-05-25T14:28:25Z")
1. I’ve deployed minio using Tenant; Service objects in k8s I’ve in minio > kubectl get svc -n minio > ``` > NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE > console ClusterIP 10.99.206.197 9090/TCP,9443/TCP 11d > minio ClusterIP 10.98.90.91 443/TCP 11d > operator ClusterIP 10.111.225.139 4222/TCP,4233/TCP 11d > minio-console ClusterIP 10.97.53.216 9443/TCP 11d > minio-hl ClusterIP None 9000/TCP 11d > > ``` 1. I was trying it with minio-hl.minio.svc.cluster.local:9000 2. I used mc command with public URL; If this error pops up > error: |- > mongodump: write data: upload to S3: RequestError: send request failed > caused by: Put “[https://minio.minio.svc.cluster.local:443/sp-test/2021-05-25T11%3A31%3A25Z\_cfg.dump.gz](https://minio.minio.svc.cluster.local:443/sp-test/2021-05-25T11%3A31%3A25Z_cfg.dump.gz)”: x509: certificate signed by unknown authority. I get this error when I use endpoint url as > [https://minio-hl.minio.svc.cluster.local:9000](https://minio-hl.minio.svc.cluster.local:9000) Where should be the certificates included; Where should I set the certificates? I didn’t find any configs where we can include certificates ---
**Author:** ![Sergey\_Pronin](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/sergey_pronin/32/14887_2.png) [@Sergey\_Pronin](https://forums.percona.com/u/Sergey_Pronin)\ **Post date:** [May 25, 2021, 7:38pm UTC](https://forums.percona.com/t/check-storage-connecion-unable-to-get-storage-get-config-get-mongo-no-documents-in-result/10482/14 "2021-05-25T19:38:27Z")
[This](https://github.com/minio/operator/blob/master/helm/minio-operator/values.yaml#L85-L96) section in values.yaml of Minio helm chart allows you to configure the certs. Example can be found here: [minio/docs/tls/kubernetes at master · minio/minio · GitHub](https://github.com/minio/minio/tree/master/docs/tls/kubernetes#2-create-kubernetes-secret) ---
**Author:** ![SpoorthiPalakshaiah](https://avatars.discourse-cdn.com/v4/letter/s/8dc957/32.png) [@SpoorthiPalakshaiah](https://forums.percona.com/u/SpoorthiPalakshaiah)\ **Post date:** [May 26, 2021, 4:17am UTC](https://forums.percona.com/t/check-storage-connecion-unable-to-get-storage-get-config-get-mongo-no-documents-in-result/10482/15 "2021-05-26T04:17:59Z")
I have certs in minio; Where should I add them in mongodb? ---
**Author:** ![Sergey\_Pronin](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/sergey_pronin/32/14887_2.png) [@Sergey\_Pronin](https://forums.percona.com/u/Sergey_Pronin)\ **Post date:** [May 26, 2021, 8:43am UTC](https://forums.percona.com/t/check-storage-connecion-unable-to-get-storage-get-config-get-mongo-no-documents-in-result/10482/16 "2021-05-26T08:43:52Z")
If these certs are not self-issued and valid - you don’t need to add them to mongodb. ---
**Author:** ![SpoorthiPalakshaiah](https://avatars.discourse-cdn.com/v4/letter/s/8dc957/32.png) [@SpoorthiPalakshaiah](https://forums.percona.com/u/SpoorthiPalakshaiah)\ **Post date:** [May 26, 2021, 9:01am UTC](https://forums.percona.com/t/check-storage-connecion-unable-to-get-storage-get-config-get-mongo-no-documents-in-result/10482/17 "2021-05-26T09:01:37Z")
Is there a way to disable cert check? ---
**Author:** ![Sergey\_Pronin](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/sergey_pronin/32/14887_2.png) [@Sergey\_Pronin](https://forums.percona.com/u/Sergey_Pronin)\ **Post date:** [May 26, 2021, 9:13am UTC](https://forums.percona.com/t/check-storage-connecion-unable-to-get-storage-get-config-get-mongo-no-documents-in-result/10482/18 "2021-05-26T09:13:15Z")
As I mentioned before - no. Underlying tool (Percona Backup for MongoDB) does not support that yet. We are going to implement that in later releases. There are two workarounds: 1. Run Minio without https (not sure how to do that with Operator) 2. Run Minio with valid public cert ---
**Author:** ![SpoorthiPalakshaiah](https://avatars.discourse-cdn.com/v4/letter/s/8dc957/32.png) [@SpoorthiPalakshaiah](https://forums.percona.com/u/SpoorthiPalakshaiah)\ **Post date:** [May 27, 2021, 4:48am UTC](https://forums.percona.com/t/check-storage-connecion-unable-to-get-storage-get-config-get-mongo-no-documents-in-result/10482/19 "2021-05-27T04:48:18Z")
When I use minio with valid public cert: .pbm.init file gets into the bucket but the backup shows error as > couldn’t get response from all shards: convergeClusterWithTimeout: reached converge timeout ---
**Author:** ![SpoorthiPalakshaiah](https://avatars.discourse-cdn.com/v4/letter/s/8dc957/32.png) [@SpoorthiPalakshaiah](https://forums.percona.com/u/SpoorthiPalakshaiah)\ **Post date:** [June 1, 2021, 7:18am UTC](https://forums.percona.com/t/check-storage-connecion-unable-to-get-storage-get-config-get-mongo-no-documents-in-result/10482/20 "2021-06-01T07:18:03Z")
@Sergey_Pronin Hi, we are running minio with valid public cert; My question is should we need to include that crt and key anywhere in mongodb configurations? [Next page](https://forums.percona.com/t/check-storage-connecion-unable-to-get-storage-get-config-get-mongo-no-documents-in-result/10482.md?page=2)