ERROR with storage: storage check failed with: get S3 object header: RequestError: send request failed

MongoDB Percona Operator for MongoDB
,
1

I’m using percona mongodb kubernetes operator
https://github.com/percona/percona-server-mongodb-operator/blob/main/deploy/cr.yaml

I’m trying to store backup in minio; But when I do so the backup is not getting stored
When I check for status using
pbm status --mongodb-uri=‘mongodb://<backup_user>:<backup_pwd>@localhost:27017/?replicaSet=rs0’

I get this error
> - rs0/mongo-db-rs0-1.mongo-db-rs0.mongodb.svc.cluster.local:27017: pbm-agent v1.4.1 FAILED status:

      > ERROR with storage: storage check failed with: get S3 object header: RequestError: send request failed
caused by: Head "https://<minio-svc-name>.<minio-ns>.svc.cluster.local:9000/<bucket_name>/.pbm.init": http: server gave HTTP response to HTTPS client

I tried to disable TLS as mentioned in [docs](Transport encryption (TLS/SSL) - Percona Operator for MongoDB

But the shard pods will not come up;

Please suggest the way such that backup with s3 minio works;

1 Like
2

Hello @SpoorthiPalakshaiah ,

are you taking the backup through psmdb-backup Custom Resource?
If so, please share the backup.storages section and backup.yaml config.
If not - please share the steps to reproduce.

Judging by the error I would say that your minio server is running as http one, but you are trying to refer to it as https. In Operator you can try to resolve this by setting endpointUrl correctly.

1 Like
3

Hello @Sergey_Pronin,
Here is the backup file and storage details
backup.yaml

apiVersion: psmdb.percona.com/v1
kind: PerconaServerMongoDBBackup
metadata:
name: mongodb-test-backup
spec:
psmdbCluster: mongo-db
storageName: mongo-backup

Storage is included in cr.yaml;

storages:
mongo-backup:
type: s3
s3:
bucket: mongodb-bkp
region: us-east-1
credentialsSecret: mongodb-cluster-backup-secret
endpointUrl: minio-svc.minio-namespace.svc.cluster.local:9000

When I try to give endpoint as http://minio-svc.minio-namespace.svc.cluster.local:9000; It gives same error

1 Like
4

@SpoorthiPalakshaiah try to set endpointUrl to “http://minio-svc.minio-namespace.svc.cluster.local:9000”. Without schema set it tries to connect via https by default.

1 Like
5

Oh, you did this already…

1 Like
6

@SpoorthiPalakshaiah After applying http://minio-svc.minio-namespace.svc.cluster.local:9000 what does pbm config shows? Has the new config applied?

1 Like
7

@Andrew_Pogrebnoi
In pbm config the endpoint shows with http://minio-svc.minio-namespace.svc.cluster.local:9000

But in pbm status it shows

2021-05-11T08:15:16Z 0.00B [ERROR: mongodump: write data: upload to S3: RequestError: send request failed
caused by: Put “https://minio-svc.minio-ns.svc.cluster.local:9000/mongodb-bkp/2021-05-11T08%3A15%3A16Z_cfg.dump.gz”: http: server gave HTTP response to HTTPS client.] [2021-05-11T08:16:01]
2021-05-11T00:00:12Z 0.00B [ERROR: some of pbm-agents were lost during the backup] [2021-05-11T00:01:07]

8

Even you change the endpointUrl in cr.yaml, it will be updated in PBM (backup-agent) container only after you try to take another backup or perform the restoration.

Could you please set endpointUrl in cr.yaml with http and try to take the backup through psmdb-backup CR?

2 Likes
9

@Sergey_Pronin When I take another backup only the .pbm.init file will be stored inside minio bucket;

And the pbm status is shown as

Backups:

S3 us-east-1 http://..svc.cluster.local:9000/mongodb-bkp
Snapshots:
2021-05-13T05:06:52Z 0.00B [ERROR: check cluster for backup started: convergeClusterWithTimeout: lost shard cfg, last beat ts: 1620882434] [2021-05-13T05:07:46]
2021-05-13T05:01:01Z 0.00B [ERROR: some of pbm-agents were lost during the backup] [2021-05-13T05:01:57]

1 Like