Hi,
We Instsall the latest version pbm 2.8.0 and we got physical backup fails on stage files cheks
2025-01-24T14:07:52Z 6.52TB <physical> [ERROR: check backup files: file 2025-01-24T14:07:52Z/rs11/collection-675--6657163989133444963.wt.zst: get S3 object header: ServiceUnavailable: Service Unavailable
status code: 503, request id: 181DE6E201977C1D, host id: ] [2025-01-25T10:05:50Z]
pbm describe-backup 2025-01-24T14:07:52Z
name: "2025-01-24T14:07:52Z"
opid: 67939eb8f92d99253ba99a80
type: physical
last_write_time: "2025-01-24T14:07:55Z"
last_transition_time: "2025-01-25T10:05:50Z"
mongodb_version: 5.0.17-14
fcv: "5.0"
pbm_version: 2.8.0
status: error
size_h: 6.5 TiB
error: "check backup files: file 2025-01-24T14:07:52Z/rs11/collection-675--6657163989133444963.wt.zst:
get S3 object header: ServiceUnavailable: Service Unavailable\n\tstatus code: 503,
request id: 181DE6E201977C1D, host id: "
replsets:
- name: rs11
status: error
node: [REDACTED]
last_write_time: "2025-01-24T14:07:55Z"
last_transition_time: "2025-01-25T10:05:50Z"
error: "check backup files: file 2025-01-24T14:07:52Z/rs11/collection-675--6657163989133444963.wt.zst:
get S3 object header: ServiceUnavailable: Service Unavailable\n\tstatus code:
503, request id: 181DE6E201977C1D, host id: "
Downgraded version to 2.7.0 and try again and got this error again. I am research changelog maybe this PBM-1397
du -sh /data/db/collection-675--6657163989133444963.wt
4.4T /data/db/collection-675--6657163989133444963.wt
Okay, we now tested s3 cluster (minio) and didn’t find any problems there. All huge files available and can be downloaded.
Another tests usage utils:
pbm-speed-test storage --compression=zstd --size-gb 1024
1.00TB sent in 27m11s.
Avg upload rate = 643.10MB/s.
Now we downgrade pbm to version 2.6.0 and physical completed successfully
2025-01-26T17:55:12Z 6.51TB <physical> [restore_to_time: 2025-01-26T17:55:15Z]
pbm describe-backup 2025-01-26T17:55:12Z
name: "2025-01-26T17:55:12Z"
opid: 6796770035bfc111fad1bfa1
type: physical
last_write_time: "2025-01-26T17:55:15Z"
last_transition_time: "2025-01-27T13:22:33Z"
mongodb_version: 5.0.17-14
fcv: "5.0"
pbm_version: 2.6.0
status: done
size_h: 6.5 TiB
replsets:
- name: rs11
status: done
node: [REDACTED]
last_write_time: "2025-01-26T17:55:15Z"
last_transition_time: "2025-01-27T13:22:32Z"
Config file
storage:
type: s3
s3:
region: minio
endpointUrl: '***'
forcePathStyle: true
bucket: test
prefix: pbm/main
credentials:
access-key-id: '***'
secret-access-key: '***'
uploadPartSize: 838860800
maxUploadParts: 10000
storageClass: STANDARD
insecureSkipTLSVerify: false
debugLogLevels: LogDebug
retryer:
numMaxRetries: 30
minRetryDelay: 30ns
maxRetryDelay: 5ns
pitr:
enabled: false
compression: zstd
In the archive I have attached a diagnostic report, logs in debug mode. Secops says redacted all ips/domain name, sorry about that
Archive gdrive