Backup files are not deleted from Azure blob storage

MongoDB Percona Operator for MongoDB
1

Description:

I noticed that all backup artifacts created by the Operator using backup.tasks are not get deleted from Azure blob keep in mind I’ve already configured the backup task to keep only 3 instances.

Steps to Reproduce:

here is my config:
backup:
enabled: true
image:
repository: percona/percona-backup-mongodb
tag: 2.5.0
resources:
limits:
cpu: “300m”
memory: “1.1G”
requests:
cpu: “300m”
memory: “1G”
storages:
azure-blob:
type: azure
azure:
container: psmdb-backup02
prefix: psmdb
credentialsSecret: azure-blob-storage-secret
endpointUrl: https://xxx.blob.core.windows.net
pitr:
enabled: true
oplogOnly: true
oplogSpanMin: 10
compressionType: gzip
compressionLevel: 6
tasks:

  • name: azure-blob-task
    enabled: true
    type: logical
    schedule: “0 * * * *”
    keep: 3
    storageName: azure-blob
    compressionType: gzip

Version:

psmdb-operator-1.17.0
psmdb-db-1.17.0
Kubernetes-1.30.0

Logs:

[If applicable, include any relevant log files or error messages]

Expected Result:

[What the user expected to see or happen before the issue occurred]

Actual Result:

[What actually happened when the user encountered the issue]

Additional Information:

[Include any additional information that could be helpful to diagnose the issue, such as browser or device information]

2

Do you have any logs? It is difficult to help diagnose without any information.

3

Hi Matthewb,

Kindly find below the operator logs.

{“level”:“info”,“ts”:1728317066.0047317,“logger”:“setup”,“msg”:“Manager starting up”,“gitCommit”:“5019408f1fe40483fc5effaf61ab3f672765b189”,“gitBranch”:“release-1-17-0”,“goVersion”:“go1.22.6”,“os”:“linux”,“arch”:“amd64”}
{“level”:“info”,“ts”:1728317066.0582178,“msg”:“server version”,“platform”:“kubernetes”,“version”:“v1.30.0”}
{“level”:“info”,“ts”:1728317066.0724545,“logger”:“controller-runtime.metrics”,“msg”:“Starting metrics server”}
{“level”:“info”,“ts”:1728317066.072528,“msg”:“starting server”,“name”:“health probe”,“addr”:“[::]:8081”}
{“level”:“info”,“ts”:1728317066.0725467,“logger”:“controller-runtime.metrics”,“msg”:“Serving metrics server”,“bindAddress”:“:8080”,“secure”:false}
I1007 16:04:26.072641 1 leaderelection.go:250] attempting to acquire leader lease psmdb-operator/08db0feb.percona.com…
I1008 09:26:15.771352 1 leaderelection.go:260] successfully acquired lease psmdb-operator/08db0feb.percona.com
{“level”:“debug”,“ts”:1728379575.7714243,“logger”:“events”,“msg”:“psmdb-operator-5645b9c5c-6bnh5_3b625dff-d7f2-46a8-8a0a-1e381f99678c became leader”,“type”:“Normal”,“object”:{“kind”:“Lease”,“namespace”:“psmdb-operator”,“name”:“08db0feb.percona.com”,“uid”:“f033ec75-2e6f-4cda-8115-a3552f290952”,“apiVersion”:“coordination.k8s.io/v1",“resourceVersion”:“339398”},“reason”:"LeaderElection”}
{“level”:“info”,“ts”:1728379575.7716029,“msg”:“Starting EventSource”,“controller”:“psmdb-controller”,“source”:“kind source: *v1.PerconaServerMongoDB”}
{“level”:“info”,“ts”:1728379575.7716336,“msg”:“Starting Controller”,“controller”:“psmdb-controller”}
{“level”:“info”,“ts”:1728379575.771647,“msg”:“Starting EventSource”,“controller”:“psmdbbackup-controller”,“source”:“kind source: *v1.PerconaServerMongoDBBackup”}
{“level”:“info”,“ts”:1728379575.771663,“msg”:“Starting EventSource”,“controller”:“psmdbbackup-controller”,“source”:“kind source: *v1.Pod”}
{“level”:“info”,“ts”:1728379575.7716672,“msg”:“Starting Controller”,“controller”:“psmdbbackup-controller”}
{“level”:“info”,“ts”:1728379575.7717512,“msg”:“Starting EventSource”,“controller”:“psmdbrestore-controller”,“source”:“kind source: *v1.PerconaServerMongoDBRestore”}
{“level”:“info”,“ts”:1728379575.7717643,“msg”:“Starting EventSource”,“controller”:“psmdbrestore-controller”,“source”:“kind source: *v1.Pod”}
{“level”:“info”,“ts”:1728379575.7717686,“msg”:“Starting Controller”,“controller”:“psmdbrestore-controller”}
{“level”:“info”,“ts”:1728379576.5750985,“msg”:“Starting workers”,“controller”:“psmdb-controller”,“worker count”:1}
{“level”:“info”,“ts”:1728379576.5789456,“msg”:“Starting workers”,“controller”:“psmdbbackup-controller”,“worker count”:1}
{“level”:“info”,“ts”:1728379576.578965,“msg”:“Starting workers”,“controller”:“psmdbrestore-controller”,“worker count”:1}
{“level”:“info”,“ts”:1728379576.6607666,“msg”:“Creating or updating backup job”,“controller”:“psmdb-controller”,“object”:{“name”:“psmdb-db”,“namespace”:“psmdb”},“namespace”:“psmdb”,“name”:“psmdb-db”,“reconcileID”:“4f344998-bfcc-47e5-842a-1f1d76f1216c”,“name”:“azure-blob-task”,“namespace”:“psmdb”,“schedule”:“0 * * * *”}
{“level”:“debug”,“ts”:1728381600.2541733,“msg”:“checking if backup is allowed”,“controller”:“psmdbbackup-controller”,“object”:{“name”:“cron-psmdb-db-20241008100000-nv5km”,“namespace”:“psmdb”},“namespace”:“psmdb”,“name”:“cron-psmdb-db-20241008100000-nv5km”,“reconcileID”:“dd8d93e1-8562-4adb-94f7-278e536c6f56”,“backup”:“psmdb-db”}
{“level”:“debug”,“ts”:1728381600.2541933,“msg”:“Checking for active jobs”,“controller”:“psmdbbackup-controller”,“object”:{“name”:“cron-psmdb-db-20241008100000-nv5km”,“namespace”:“psmdb”},“namespace”:“psmdb”,“name”:“cron-psmdb-db-20241008100000-nv5km”,“reconcileID”:“dd8d93e1-8562-4adb-94f7-278e536c6f56”,“currentJob”:{“Name”:“cron-psmdb-db-20241008100000-nv5km”,“Type”:0}}
{“level”:“info”,“ts”:1728381610.2777383,“msg”:“Starting backup”,“controller”:“psmdbbackup-controller”,“object”:{“name”:“cron-psmdb-db-20241008100000-nv5km”,“namespace”:“psmdb”},“namespace”:“psmdb”,“name”:“cron-psmdb-db-20241008100000-nv5km”,“reconcileID”:“dd8d93e1-8562-4adb-94f7-278e536c6f56”,“backup”:“cron-psmdb-db-20241008100000-nv5km”,“storage”:“azure-blob”}
{“level”:“info”,“ts”:1728381610.2778616,“msg”:“Setting PBM config”,“controller”:“psmdbbackup-controller”,“object”:{“name”:“cron-psmdb-db-20241008100000-nv5km”,“namespace”:“psmdb”},“namespace”:“psmdb”,“name”:“cron-psmdb-db-20241008100000-nv5km”,“reconcileID”:“dd8d93e1-8562-4adb-94f7-278e536c6f56”,“backup”:“psmdb-db”}
{“level”:“info”,“ts”:1728381615.2698371,“msg”:“Setting pitr.oplogOnly in PBM config”,“controller”:“psmdb-controller”,“object”:{“name”:“psmdb-db”,“namespace”:“psmdb”},“namespace”:“psmdb”,“name”:“psmdb-db”,“reconcileID”:“9f5c12b1-9b62-4725-a585-319d0d65f46d”,“value”:“true”}
{“level”:“info”,“ts”:1728381621.2838626,“msg”:“Sending backup command”,“controller”:“psmdbbackup-controller”,“object”:{“name”:“cron-psmdb-db-20241008100000-nv5km”,“namespace”:“psmdb”},“namespace”:“psmdb”,“name”:“cron-psmdb-db-20241008100000-nv5km”,“reconcileID”:“dd8d93e1-8562-4adb-94f7-278e536c6f56”,“backupCmd”:“backup [name: 2024-10-08T10:00:21Z, compression: gzip (level: default)] <ts: 0>”}
{“level”:“debug”,“ts”:1728381621.3113432,“msg”:“checking if backup is allowed”,“controller”:“psmdbbackup-controller”,“object”:{“name”:“cron-psmdb-db-20241008100000-nv5km”,“namespace”:“psmdb”},“namespace”:“psmdb”,“name”:“cron-psmdb-db-20241008100000-nv5km”,“reconcileID”:“1804947e-02df-4e6e-8fb0-d46b1963fc21”,“backup”:“psmdb-db”}
{“level”:“debug”,“ts”:1728381621.3113587,“msg”:“Checking for active jobs”,“controller”:“psmdbbackup-controller”,“object”:{“name”:“cron-psmdb-db-20241008100000-nv5km”,“namespace”:“psmdb”},“namespace”:“psmdb”,“name”:“cron-psmdb-db-20241008100000-nv5km”,“reconcileID”:“1804947e-02df-4e6e-8fb0-d46b1963fc21”,“currentJob”:{“Name”:“cron-psmdb-db-20241008100000-nv5km”,“Type”:0}}
{“level”:“debug”,“ts”:1728381626.3548427,“msg”:“checking if backup is allowed”,“controller”:“psmdbbackup-controller”,“object”:{“name”:“cron-psmdb-db-20241008100000-nv5km”,“namespace”:“psmdb”},“namespace”:“psmdb”,“name”:“cron-psmdb-db-20241008100000-nv5km”,“reconcileID”:“a31387dd-0e6e-4ca8-b0e1-16e79c312b9e”,“backup”:“psmdb-db”}
{“level”:“debug”,“ts”:1728381626.354862,“msg”:“Checking for active jobs”,“controller”:“psmdbbackup-controller”,“object”:{“name”:“cron-psmdb-db-20241008100000-nv5km”,“namespace”:“psmdb”},“namespace”:“psmdb”,“name”:“cron-psmdb-db-20241008100000-nv5km”,“reconcileID”:“a31387dd-0e6e-4ca8-b0e1-16e79c312b9e”,“currentJob”:{“Name”:“cron-psmdb-db-20241008100000-nv5km”,“Type”:0}}
{“level”:“debug”,“ts”:1728385200.0396354,“msg”:“checking if backup is allowed”,“controller”:“psmdbbackup-controller”,“object”:{“name”:“cron-psmdb-db-20241008110000-wk4fq”,“namespace”:“psmdb”},“namespace”:“psmdb”,“name”:“cron-psmdb-db-20241008110000-wk4fq”,“reconcileID”:“3b49ed00-72f3-4933-9683-32b5b0543eeb”,“backup”:“psmdb-db”}
{“level”:“debug”,“ts”:1728385200.0396602,“msg”:“Checking for active jobs”,“controller”:“psmdbbackup-controller”,“object”:{“name”:“cron-psmdb-db-20241008110000-wk4fq”,“namespace”:“psmdb”},“namespace”:“psmdb”,“name”:“cron-psmdb-db-20241008110000-wk4fq”,“reconcileID”:“3b49ed00-72f3-4933-9683-32b5b0543eeb”,“currentJob”:{“Name”:“cron-psmdb-db-20241008110000-wk4fq”,“Type”:0}}
{“level”:“info”,“ts”:1728385210.051305,“msg”:“Starting backup”,“controller”:“psmdbbackup-controller”,“object”:{“name”:“cron-psmdb-db-20241008110000-wk4fq”,“namespace”:“psmdb”},“namespace”:“psmdb”,“name”:“cron-psmdb-db-20241008110000-wk4fq”,“reconcileID”:“3b49ed00-72f3-4933-9683-32b5b0543eeb”,“backup”:“cron-psmdb-db-20241008110000-wk4fq”,“storage”:“azure-blob”}
{“level”:“info”,“ts”:1728385210.0513575,“msg”:“Setting PBM config”,“controller”:“psmdbbackup-controller”,“object”:{“name”:“cron-psmdb-db-20241008110000-wk4fq”,“namespace”:“psmdb”},“namespace”:“psmdb”,“name”:“cron-psmdb-db-20241008110000-wk4fq”,“reconcileID”:“3b49ed00-72f3-4933-9683-32b5b0543eeb”,“backup”:“psmdb-db”}
{“level”:“info”,“ts”:1728385214.0950224,“msg”:“Setting pitr.oplogOnly in PBM config”,“controller”:“psmdb-controller”,“object”:{“name”:“psmdb-db”,“namespace”:“psmdb”},“namespace”:“psmdb”,“name”:“psmdb-db”,“reconcileID”:“96761318-c14b-4961-b8cd-3c8af6398e06”,“value”:“true”}
{“level”:“info”,“ts”:1728385221.057826,“msg”:“Sending backup command”,“controller”:“psmdbbackup-controller”,“object”:{“name”:“cron-psmdb-db-20241008110000-wk4fq”,“namespace”:“psmdb”},“namespace”:“psmdb”,“name”:“cron-psmdb-db-20241008110000-wk4fq”,“reconcileID”:“3b49ed00-72f3-4933-9683-32b5b0543eeb”,“backupCmd”:“backup [name: 2024-10-08T11:00:21Z, compression: gzip (level: default)] <ts: 0>”}
{“level”:“debug”,“ts”:1728385221.0869524,“msg”:“checking if backup is allowed”,“controller”:“psmdbbackup-controller”,“object”:{“name”:“cron-psmdb-db-20241008110000-wk4fq”,“namespace”:“psmdb”},“namespace”:“psmdb”,“name”:“cron-psmdb-db-20241008110000-wk4fq”,“reconcileID”:“f26e5b2d-345e-42aa-8798-13272a7e8cda”,“backup”:“psmdb-db”}
{“level”:“debug”,“ts”:1728385221.086966,“msg”:“Checking for active jobs”,“controller”:“psmdbbackup-controller”,“object”:{“name”:“cron-psmdb-db-20241008110000-wk4fq”,“namespace”:“psmdb”},“namespace”:“psmdb”,“name”:“cron-psmdb-db-20241008110000-wk4fq”,“reconcileID”:“f26e5b2d-345e-42aa-8798-13272a7e8cda”,“currentJob”:{“Name”:“cron-psmdb-db-20241008110000-wk4fq”,“Type”:0}}
{“level”:“debug”,“ts”:1728385226.129725,“msg”:“checking if backup is allowed”,“controller”:“psmdbbackup-controller”,“object”:{“name”:“cron-psmdb-db-20241008110000-wk4fq”,“namespace”:“psmdb”},“namespace”:“psmdb”,“name”:“cron-psmdb-db-20241008110000-wk4fq”,“reconcileID”:“237ad055-8bec-4251-bec0-6a0f02d5002b”,“backup”:“psmdb-db”}
{“level”:“debug”,“ts”:1728385226.129748,“msg”:“Checking for active jobs”,“controller”:“psmdbbackup-controller”,“object”:{“name”:“cron-psmdb-db-20241008110000-wk4fq”,“namespace”:“psmdb”},“namespace”:“psmdb”,“name”:“cron-psmdb-db-20241008110000-wk4fq”,“reconcileID”:“237ad055-8bec-4251-bec0-6a0f02d5002b”,“currentJob”:{“Name”:“cron-psmdb-db-20241008110000-wk4fq”,“Type”:0}}
{“level”:“debug”,“ts”:1728388800.040303,“msg”:“checking if backup is allowed”,“controller”:“psmdbbackup-controller”,“object”:{“name”:“cron-psmdb-db-20241008120000-z54v2”,“namespace”:“psmdb”},“namespace”:“psmdb”,“name”:“cron-psmdb-db-20241008120000-z54v2”,“reconcileID”:“1577bd4a-c3b9-4d03-8eaa-2b222a847306”,“backup”:“psmdb-db”}
{“level”:“debug”,“ts”:1728388800.0403223,“msg”:“Checking for active jobs”,“controller”:“psmdbbackup-controller”,“object”:{“name”:“cron-psmdb-db-20241008120000-z54v2”,“namespace”:“psmdb”},“namespace”:“psmdb”,“name”:“cron-psmdb-db-20241008120000-z54v2”,“reconcileID”:“1577bd4a-c3b9-4d03-8eaa-2b222a847306”,“currentJob”:{“Name”:“cron-psmdb-db-20241008120000-z54v2”,“Type”:0}}
{“level”:“info”,“ts”:1728388810.0646815,“msg”:“Starting backup”,“controller”:“psmdbbackup-controller”,“object”:{“name”:“cron-psmdb-db-20241008120000-z54v2”,“namespace”:“psmdb”},“namespace”:“psmdb”,“name”:“cron-psmdb-db-20241008120000-z54v2”,“reconcileID”:“1577bd4a-c3b9-4d03-8eaa-2b222a847306”,“backup”:“cron-psmdb-db-20241008120000-z54v2”,“storage”:“azure-blob”}
{“level”:“info”,“ts”:1728388810.0647388,“msg”:“Setting PBM config”,“controller”:“psmdbbackup-controller”,“object”:{“name”:“cron-psmdb-db-20241008120000-z54v2”,“namespace”:“psmdb”},“namespace”:“psmdb”,“name”:“cron-psmdb-db-20241008120000-z54v2”,“reconcileID”:“1577bd4a-c3b9-4d03-8eaa-2b222a847306”,“backup”:“psmdb-db”}
{“level”:“info”,“ts”:1728388812.4031556,“msg”:“Setting pitr.oplogOnly in PBM config”,“controller”:“psmdb-controller”,“object”:{“name”:“psmdb-db”,“namespace”:“psmdb”},“namespace”:“psmdb”,“name”:“psmdb-db”,“reconcileID”:“ce63a674-6456-4364-bbc2-937abab9bd49”,“value”:“true”}
{“level”:“info”,“ts”:1728388821.070425,“msg”:“Sending backup command”,“controller”:“psmdbbackup-controller”,“object”:{“name”:“cron-psmdb-db-20241008120000-z54v2”,“namespace”:“psmdb”},“namespace”:“psmdb”,“name”:“cron-psmdb-db-20241008120000-z54v2”,“reconcileID”:“1577bd4a-c3b9-4d03-8eaa-2b222a847306”,“backupCmd”:“backup [name: 2024-10-08T12:00:21Z, compression: gzip (level: default)] <ts: 0>”}
{“level”:“debug”,“ts”:1728388821.0972238,“msg”:“checking if backup is allowed”,“controller”:“psmdbbackup-controller”,“object”:{“name”:“cron-psmdb-db-20241008120000-z54v2”,“namespace”:“psmdb”},“namespace”:“psmdb”,“name”:“cron-psmdb-db-20241008120000-z54v2”,“reconcileID”:“c9504129-9d3a-42c4-af17-d01e449ea358”,“backup”:“psmdb-db”}
{“level”:“debug”,“ts”:1728388821.097241,“msg”:“Checking for active jobs”,“controller”:“psmdbbackup-controller”,“object”:{“name”:“cron-psmdb-db-20241008120000-z54v2”,“namespace”:“psmdb”},“namespace”:“psmdb”,“name”:“cron-psmdb-db-20241008120000-z54v2”,“reconcileID”:“c9504129-9d3a-42c4-af17-d01e449ea358”,“currentJob”:{“Name”:“cron-psmdb-db-20241008120000-z54v2”,“Type”:0}}
{“level”:“debug”,“ts”:1728388826.1431954,“msg”:“checking if backup is allowed”,“controller”:“psmdbbackup-controller”,“object”:{“name”:“cron-psmdb-db-20241008120000-z54v2”,“namespace”:“psmdb”},“namespace”:“psmdb”,“name”:“cron-psmdb-db-20241008120000-z54v2”,“reconcileID”:“d7d6f0dc-9a03-4aa5-aa01-38772b9ec021”,“backup”:“psmdb-db”}
{“level”:“debug”,“ts”:1728388826.1432137,“msg”:“Checking for active jobs”,“controller”:“psmdbbackup-controller”,“object”:{“name”:“cron-psmdb-db-20241008120000-z54v2”,“namespace”:“psmdb”},“namespace”:“psmdb”,“name”:“cron-psmdb-db-20241008120000-z54v2”,“reconcileID”:“d7d6f0dc-9a03-4aa5-aa01-38772b9ec021”,“currentJob”:{“Name”:“cron-psmdb-db-20241008120000-z54v2”,“Type”:0}}
{“level”:“debug”,“ts”:1728392400.0423682,“msg”:“checking if backup is allowed”,“controller”:“psmdbbackup-controller”,“object”:{“name”:“cron-psmdb-db-20241008130000-rbhns”,“namespace”:“psmdb”},“namespace”:“psmdb”,“name”:“cron-psmdb-db-20241008130000-rbhns”,“reconcileID”:“6abf54e0-0475-41ca-af66-f00cd080631c”,“backup”:“psmdb-db”}
{“level”:“debug”,“ts”:1728392400.042388,“msg”:“Checking for active jobs”,“controller”:“psmdbbackup-controller”,“object”:{“name”:“cron-psmdb-db-20241008130000-rbhns”,“namespace”:“psmdb”},“namespace”:“psmdb”,“name”:“cron-psmdb-db-20241008130000-rbhns”,“reconcileID”:“6abf54e0-0475-41ca-af66-f00cd080631c”,“currentJob”:{“Name”:“cron-psmdb-db-20241008130000-rbhns”,“Type”:0}}
{“level”:“info”,“ts”:1728392410.0558026,“msg”:“Starting backup”,“controller”:“psmdbbackup-controller”,“object”:{“name”:“cron-psmdb-db-20241008130000-rbhns”,“namespace”:“psmdb”},“namespace”:“psmdb”,“name”:“cron-psmdb-db-20241008130000-rbhns”,“reconcileID”:“6abf54e0-0475-41ca-af66-f00cd080631c”,“backup”:“cron-psmdb-db-20241008130000-rbhns”,“storage”:“azure-blob”}
{“level”:“info”,“ts”:1728392410.055841,“msg”:“Setting PBM config”,“controller”:“psmdbbackup-controller”,“object”:{“name”:“cron-psmdb-db-20241008130000-rbhns”,“namespace”:“psmdb”},“namespace”:“psmdb”,“name”:“cron-psmdb-db-20241008130000-rbhns”,“reconcileID”:“6abf54e0-0475-41ca-af66-f00cd080631c”,“backup”:“psmdb-db”}
{“level”:“info”,“ts”:1728392410.4644418,“msg”:“Setting pitr.oplogOnly in PBM config”,“controller”:“psmdb-controller”,“object”:{“name”:“psmdb-db”,“namespace”:“psmdb”},“namespace”:“psmdb”,“name”:“psmdb-db”,“reconcileID”:“6809a977-69cb-4e58-92b7-830a13e5675e”,“value”:“true”}
{“level”:“info”,“ts”:1728392421.0611565,“msg”:“Sending backup command”,“controller”:“psmdbbackup-controller”,“object”:{“name”:“cron-psmdb-db-20241008130000-rbhns”,“namespace”:“psmdb”},“namespace”:“psmdb”,“name”:“cron-psmdb-db-20241008130000-rbhns”,“reconcileID”:“6abf54e0-0475-41ca-af66-f00cd080631c”,“backupCmd”:“backup [name: 2024-10-08T13:00:21Z, compression: gzip (level: default)] <ts: 0>”}
{“level”:“debug”,“ts”:1728392421.0907848,“msg”:“checking if backup is allowed”,“controller”:“psmdbbackup-controller”,“object”:{“name”:“cron-psmdb-db-20241008130000-rbhns”,“namespace”:“psmdb”},“namespace”:“psmdb”,“name”:“cron-psmdb-db-20241008130000-rbhns”,“reconcileID”:“54bc21a9-43e2-4b48-8b79-f6aacd19930b”,“backup”:“psmdb-db”}
{“level”:“debug”,“ts”:1728392421.0908,“msg”:“Checking for active jobs”,“controller”:“psmdbbackup-controller”,“object”:{“name”:“cron-psmdb-db-20241008130000-rbhns”,“namespace”:“psmdb”},“namespace”:“psmdb”,“name”:“cron-psmdb-db-20241008130000-rbhns”,“reconcileID”:“54bc21a9-43e2-4b48-8b79-f6aacd19930b”,“currentJob”:{“Name”:“cron-psmdb-db-20241008130000-rbhns”,“Type”:0}}
{“level”:“debug”,“ts”:1728392426.14336,“msg”:“checking if backup is allowed”,“controller”:“psmdbbackup-controller”,“object”:{“name”:“cron-psmdb-db-20241008130000-rbhns”,“namespace”:“psmdb”},“namespace”:“psmdb”,“name”:“cron-psmdb-db-20241008130000-rbhns”,“reconcileID”:“26a9f711-de76-4e53-b4c9-f9c7d2409adc”,“backup”:“psmdb-db”}
{“level”:“debug”,“ts”:1728392426.1433833,“msg”:“Checking for active jobs”,“controller”:“psmdbbackup-controller”,“object”:{“name”:“cron-psmdb-db-20241008130000-rbhns”,“namespace”:“psmdb”},“namespace”:“psmdb”,“name”:“cron-psmdb-db-20241008130000-rbhns”,“reconcileID”:“26a9f711-de76-4e53-b4c9-f9c7d2409adc”,“currentJob”:{“Name”:“cron-psmdb-db-20241008130000-rbhns”,“Type”:0}}
{“level”:“debug”,“ts”:1728396000.041515,“msg”:“checking if backup is allowed”,“controller”:“psmdbbackup-controller”,“object”:{“name”:“cron-psmdb-db-20241008140000-ws2q5”,“namespace”:“psmdb”},“namespace”:“psmdb”,“name”:“cron-psmdb-db-20241008140000-ws2q5”,“reconcileID”:“6de31065-e34e-49cb-8ac6-fc55db40a7c7”,“backup”:“psmdb-db”}
{“level”:“debug”,“ts”:1728396000.0415332,“msg”:“Checking for active jobs”,“controller”:“psmdbbackup-controller”,“object”:{“name”:“cron-psmdb-db-20241008140000-ws2q5”,“namespace”:“psmdb”},“namespace”:“psmdb”,“name”:“cron-psmdb-db-20241008140000-ws2q5”,“reconcileID”:“6de31065-e34e-49cb-8ac6-fc55db40a7c7”,“currentJob”:{“Name”:“cron-psmdb-db-20241008140000-ws2q5”,“Type”:0}}
{“level”:“info”,“ts”:1728396010.0815394,“msg”:“Starting backup”,“controller”:“psmdbbackup-controller”,“object”:{“name”:“cron-psmdb-db-20241008140000-ws2q5”,“namespace”:“psmdb”},“namespace”:“psmdb”,“name”:“cron-psmdb-db-20241008140000-ws2q5”,“reconcileID”:“6de31065-e34e-49cb-8ac6-fc55db40a7c7”,“backup”:“cron-psmdb-db-20241008140000-ws2q5”,“storage”:“azure-blob”}
{“level”:“info”,“ts”:1728396010.0816095,“msg”:“Setting PBM config”,“controller”:“psmdbbackup-controller”,“object”:{“name”:“cron-psmdb-db-20241008140000-ws2q5”,“namespace”:“psmdb”},“namespace”:“psmdb”,“name”:“cron-psmdb-db-20241008140000-ws2q5”,“reconcileID”:“6de31065-e34e-49cb-8ac6-fc55db40a7c7”,“backup”:“psmdb-db”}
{“level”:“info”,“ts”:1728396014.1694424,“msg”:“Setting pitr.oplogOnly in PBM config”,“controller”:“psmdb-controller”,“object”:{“name”:“psmdb-db”,“namespace”:“psmdb”},“namespace”:“psmdb”,“name”:“psmdb-db”,“reconcileID”:“b9f73b03-f45e-49a6-a679-1a7cd2d9a188”,“value”:“true”}
{“level”:“info”,“ts”:1728396021.08646,“msg”:“Sending backup command”,“controller”:“psmdbbackup-controller”,“object”:{“name”:“cron-psmdb-db-20241008140000-ws2q5”,“namespace”:“psmdb”},“namespace”:“psmdb”,“name”:“cron-psmdb-db-20241008140000-ws2q5”,“reconcileID”:“6de31065-e34e-49cb-8ac6-fc55db40a7c7”,“backupCmd”:“backup [name: 2024-10-08T14:00:21Z, compression: gzip (level: default)] <ts: 0>”}

4

{“level”:“debug”,“ts”:1728428421.116063,“msg”:“checking if backup is allowed”,“controller”:“psmdbbackup-controller”,“object”:{“name”:“cron-psmdb-db-20241008230000-pgv98”,“namespace”:“psmdb”},“namespace”:“psmdb”,“name”:“cron-psmdb-db-20241008230000-pgv98”,“reconcileID”:“caf6c302-8a7b-46e3-b8e0-2daa6147525c”,“backup”:“psmdb-db”}
{“level”:“debug”,“ts”:1728428421.1160815,“msg”:“Checking for active jobs”,“controller”:“psmdbbackup-controller”,“object”:{“name”:“cron-psmdb-db-20241008230000-pgv98”,“namespace”:“psmdb”},“namespace”:“psmdb”,“name”:“cron-psmdb-db-20241008230000-pgv98”,“reconcileID”:“caf6c302-8a7b-46e3-b8e0-2daa6147525c”,“currentJob”:{“Name”:“cron-psmdb-db-20241008230000-pgv98”,“Type”:0}}
{“level”:“debug”,“ts”:1728428426.1548321,“msg”:“checking if backup is allowed”,“controller”:“psmdbbackup-controller”,“object”:{“name”:“cron-psmdb-db-20241008230000-pgv98”,“namespace”:“psmdb”},“namespace”:“psmdb”,“name”:“cron-psmdb-db-20241008230000-pgv98”,“reconcileID”:“1e1591ea-5a70-49e8-ac28-c19035cf804b”,“backup”:“psmdb-db”}
{“level”:“debug”,“ts”:1728428426.1548498,“msg”:“Checking for active jobs”,“controller”:“psmdbbackup-controller”,“object”:{“name”:“cron-psmdb-db-20241008230000-pgv98”,“namespace”:“psmdb”},“namespace”:“psmdb”,“name”:“cron-psmdb-db-20241008230000-pgv98”,“reconcileID”:“1e1591ea-5a70-49e8-ac28-c19035cf804b”,“currentJob”:{“Name”:“cron-psmdb-db-20241008230000-pgv98”,“Type”:0}}
{“level”:“debug”,“ts”:1728432000.0493152,“msg”:“checking if backup is allowed”,“controller”:“psmdbbackup-controller”,“object”:{“name”:“cron-psmdb-db-20241009000000-wf9jk”,“namespace”:“psmdb”},“namespace”:“psmdb”,“name”:“cron-psmdb-db-20241009000000-wf9jk”,“reconcileID”:“839668a8-8e02-42cd-ab8f-1b1eef3a7944”,“backup”:“psmdb-db”}
{“level”:“debug”,“ts”:1728432000.0493383,“msg”:“Checking for active jobs”,“controller”:“psmdbbackup-controller”,“object”:{“name”:“cron-psmdb-db-20241009000000-wf9jk”,“namespace”:“psmdb”},“namespace”:“psmdb”,“name”:“cron-psmdb-db-20241009000000-wf9jk”,“reconcileID”:“839668a8-8e02-42cd-ab8f-1b1eef3a7944”,“currentJob”:{“Name”:“cron-psmdb-db-20241009000000-wf9jk”,“Type”:0}}
{“level”:“info”,“ts”:1728432010.0721664,“msg”:“Starting backup”,“controller”:“psmdbbackup-controller”,“object”:{“name”:“cron-psmdb-db-20241009000000-wf9jk”,“namespace”:“psmdb”},“namespace”:“psmdb”,“name”:“cron-psmdb-db-20241009000000-wf9jk”,“reconcileID”:“839668a8-8e02-42cd-ab8f-1b1eef3a7944”,“backup”:“cron-psmdb-db-20241009000000-wf9jk”,“storage”:“azure-blob”}
{“level”:“info”,“ts”:1728432010.0722187,“msg”:“Setting PBM config”,“controller”:“psmdbbackup-controller”,“object”:{“name”:“cron-psmdb-db-20241009000000-wf9jk”,“namespace”:“psmdb”},“namespace”:“psmdb”,“name”:“cron-psmdb-db-20241009000000-wf9jk”,“reconcileID”:“839668a8-8e02-42cd-ab8f-1b1eef3a7944”,“backup”:“psmdb-db”}
{“level”:“info”,“ts”:1728432012.3982573,“msg”:“Setting pitr.oplogOnly in PBM config”,“controller”:“psmdb-controller”,“object”:{“name”:“psmdb-db”,“namespace”:“psmdb”},“namespace”:“psmdb”,“name”:“psmdb-db”,“reconcileID”:“dbc4f80f-94ab-47da-816d-67e552d0f695”,“value”:“true”}
{“level”:“info”,“ts”:1728432021.0774157,“msg”:“Sending backup command”,“controller”:“psmdbbackup-controller”,“object”:{“name”:“cron-psmdb-db-20241009000000-wf9jk”,“namespace”:“psmdb”},“namespace”:“psmdb”,“name”:“cron-psmdb-db-20241009000000-wf9jk”,“reconcileID”:“839668a8-8e02-42cd-ab8f-1b1eef3a7944”,“backupCmd”:“backup [name: 2024-10-09T00:00:21Z, compression: gzip (level: default)] <ts: 0>”}
{“level”:“debug”,“ts”:1728432021.1089356,“msg”:“checking if backup is allowed”,“controller”:“psmdbbackup-controller”,“object”:{“name”:“cron-psmdb-db-20241009000000-wf9jk”,“namespace”:“psmdb”},“namespace”:“psmdb”,“name”:“cron-psmdb-db-20241009000000-wf9jk”,“reconcileID”:“5d36262d-9754-4c31-b85c-69ce8b5534a4”,“backup”:“psmdb-db”}
{“level”:“debug”,“ts”:1728432021.108951,“msg”:“Checking for active jobs”,“controller”:“psmdbbackup-controller”,“object”:{“name”:“cron-psmdb-db-20241009000000-wf9jk”,“namespace”:“psmdb”},“namespace”:“psmdb”,“name”:“cron-psmdb-db-20241009000000-wf9jk”,“reconcileID”:“5d36262d-9754-4c31-b85c-69ce8b5534a4”,“currentJob”:{“Name”:“cron-psmdb-db-20241009000000-wf9jk”,“Type”:0}}
{“level”:“debug”,“ts”:1728432026.1588552,“msg”:“checking if backup is allowed”,“controller”:“psmdbbackup-controller”,“object”:{“name”:“cron-psmdb-db-20241009000000-wf9jk”,“namespace”:“psmdb”},“namespace”:“psmdb”,“name”:“cron-psmdb-db-20241009000000-wf9jk”,“reconcileID”:“7b5a5dde-af55-4055-abaf-e31c5cba8300”,“backup”:“psmdb-db”}
{“level”:“debug”,“ts”:1728432026.1588917,“msg”:“Checking for active jobs”,“controller”:“psmdbbackup-controller”,“object”:{“name”:“cron-psmdb-db-20241009000000-wf9jk”,“namespace”:“psmdb”},“namespace”:“psmdb”,“name”:“cron-psmdb-db-20241009000000-wf9jk”,“reconcileID”:“7b5a5dde-af55-4055-abaf-e31c5cba8300”,“currentJob”:{“Name”:“cron-psmdb-db-20241009000000-wf9jk”,“Type”:0}}
{“level”:“debug”,“ts”:1728435600.0520346,“msg”:“checking if backup is allowed”,“controller”:“psmdbbackup-controller”,“object”:{“name”:“cron-psmdb-db-20241009010000-7sfnk”,“namespace”:“psmdb”},“namespace”:“psmdb”,“name”:“cron-psmdb-db-20241009010000-7sfnk”,“reconcileID”:“ea45d6c2-785c-4a04-aa87-0dea5d2df3f7”,“backup”:“psmdb-db”}
{“level”:“debug”,“ts”:1728435600.0520556,“msg”:“Checking for active jobs”,“controller”:“psmdbbackup-controller”,“object”:{“name”:“cron-psmdb-db-20241009010000-7sfnk”,“namespace”:“psmdb”},“namespace”:“psmdb”,“name”:“cron-psmdb-db-20241009010000-7sfnk”,“reconcileID”:“ea45d6c2-785c-4a04-aa87-0dea5d2df3f7”,“currentJob”:{“Name”:“cron-psmdb-db-20241009010000-7sfnk”,“Type”:0}}
{“level”:“info”,“ts”:1728435610.079931,“msg”:“Starting backup”,“controller”:“psmdbbackup-controller”,“object”:{“name”:“cron-psmdb-db-20241009010000-7sfnk”,“namespace”:“psmdb”},“namespace”:“psmdb”,“name”:“cron-psmdb-db-20241009010000-7sfnk”,“reconcileID”:“ea45d6c2-785c-4a04-aa87-0dea5d2df3f7”,“backup”:“cron-psmdb-db-20241009010000-7sfnk”,“storage”:“azure-blob”}
{“level”:“info”,“ts”:1728435610.0799663,“msg”:“Setting PBM config”,“controller”:“psmdbbackup-controller”,“object”:{“name”:“cron-psmdb-db-20241009010000-7sfnk”,“namespace”:“psmdb”},“namespace”:“psmdb”,“name”:“cron-psmdb-db-20241009010000-7sfnk”,“reconcileID”:“ea45d6c2-785c-4a04-aa87-0dea5d2df3f7”,“backup”:“psmdb-db”}
{“level”:“info”,“ts”:1728435610.3437142,“msg”:“Setting pitr.oplogOnly in PBM config”,“controller”:“psmdb-controller”,“object”:{“name”:“psmdb-db”,“namespace”:“psmdb”},“namespace”:“psmdb”,“name”:“psmdb-db”,“reconcileID”:“8f6f9682-708e-4bf2-a50d-738e9c973437”,“value”:“true”}
{“level”:“info”,“ts”:1728435621.084564,“msg”:“Sending backup command”,“controller”:“psmdbbackup-controller”,“object”:{“name”:“cron-psmdb-db-20241009010000-7sfnk”,“namespace”:“psmdb”},“namespace”:“psmdb”,“name”:“cron-psmdb-db-20241009010000-7sfnk”,“reconcileID”:“ea45d6c2-785c-4a04-aa87-0dea5d2df3f7”,“backupCmd”:“backup [name: 2024-10-09T01:00:21Z, compression: gzip (level: default)] <ts: 0>”}
{“level”:“debug”,“ts”:1728435621.1142595,“msg”:“checking if backup is allowed”,“controller”:“psmdbbackup-controller”,“object”:{“name”:“cron-psmdb-db-20241009010000-7sfnk”,“namespace”:“psmdb”},“namespace”:“psmdb”,“name”:“cron-psmdb-db-20241009010000-7sfnk”,“reconcileID”:“2cc3c30b-4239-491c-a135-9367376981be”,“backup”:“psmdb-db”}
{“level”:“debug”,“ts”:1728435621.1142745,“msg”:“Checking for active jobs”,“controller”:“psmdbbackup-controller”,“object”:{“name”:“cron-psmdb-db-20241009010000-7sfnk”,“namespace”:“psmdb”},“namespace”:“psmdb”,“name”:“cron-psmdb-db-20241009010000-7sfnk”,“reconcileID”:“2cc3c30b-4239-491c-a135-9367376981be”,“currentJob”:{“Name”:“cron-psmdb-db-20241009010000-7sfnk”,“Type”:0}}
{“level”:“debug”,“ts”:1728435626.1542394,“msg”:“checking if backup is allowed”,“controller”:“psmdbbackup-controller”,“object”:{“name”:“cron-psmdb-db-20241009010000-7sfnk”,“namespace”:“psmdb”},“namespace”:“psmdb”,“name”:“cron-psmdb-db-20241009010000-7sfnk”,“reconcileID”:“ce7659be-1115-4edd-b136-964fec0d84f4”,“backup”:“psmdb-db”}
{“level”:“debug”,“ts”:1728435626.1542583,“msg”:“Checking for active jobs”,“controller”:“psmdbbackup-controller”,“object”:{“name”:“cron-psmdb-db-20241009010000-7sfnk”,“namespace”:“psmdb”},“namespace”:“psmdb”,“name”:“cron-psmdb-db-20241009010000-7sfnk”,“reconcileID”:“ce7659be-1115-4edd-b136-964fec0d84f4”,“currentJob”:{“Name”:“cron-psmdb-db-20241009010000-7sfnk”,“Type”:0}}
{“level”:“debug”,“ts”:1728439200.0350583,“msg”:“checking if backup is allowed”,“controller”:“psmdbbackup-controller”,“object”:{“name”:“cron-psmdb-db-20241009020000-8t55g”,“namespace”:“psmdb”},“namespace”:“psmdb”,“name”:“cron-psmdb-db-20241009020000-8t55g”,“reconcileID”:“331b0f0e-e0c6-4ffb-b35c-b6400f9a7934”,“backup”:“psmdb-db”}
{“level”:“debug”,“ts”:1728439200.0350802,“msg”:“Checking for active jobs”,“controller”:“psmdbbackup-controller”,“object”:{“name”:“cron-psmdb-db-20241009020000-8t55g”,“namespace”:“psmdb”},“namespace”:“psmdb”,“name”:“cron-psmdb-db-20241009020000-8t55g”,“reconcileID”:“331b0f0e-e0c6-4ffb-b35c-b6400f9a7934”,“currentJob”:{“Name”:“cron-psmdb-db-20241009020000-8t55g”,“Type”:0}}
{“level”:“info”,“ts”:1728439210.0502293,“msg”:“Starting backup”,“controller”:“psmdbbackup-controller”,“object”:{“name”:“cron-psmdb-db-20241009020000-8t55g”,“namespace”:“psmdb”},“namespace”:“psmdb”,“name”:“cron-psmdb-db-20241009020000-8t55g”,“reconcileID”:“331b0f0e-e0c6-4ffb-b35c-b6400f9a7934”,“backup”:“cron-psmdb-db-20241009020000-8t55g”,“storage”:“azure-blob”}
{“level”:“info”,“ts”:1728439210.0503037,“msg”:“Setting PBM config”,“controller”:“psmdbbackup-controller”,“object”:{“name”:“cron-psmdb-db-20241009020000-8t55g”,“namespace”:“psmdb”},“namespace”:“psmdb”,“name”:“cron-psmdb-db-20241009020000-8t55g”,“reconcileID”:“331b0f0e-e0c6-4ffb-b35c-b6400f9a7934”,“backup”:“psmdb-db”}
{“level”:“info”,“ts”:1728439213.9130657,“msg”:“Setting pitr.oplogOnly in PBM config”,“controller”:“psmdb-controller”,“object”:{“name”:“psmdb-db”,“namespace”:“psmdb”},“namespace”:“psmdb”,“name”:“psmdb-db”,“reconcileID”:“d4614e52-13b4-4718-abb4-a6eb75a4e2f6”,“value”:“true”}
{“level”:“info”,“ts”:1728439221.0553632,“msg”:“Sending backup command”,“controller”:“psmdbbackup-controller”,“object”:{“name”:“cron-psmdb-db-20241009020000-8t55g”,“namespace”:“psmdb”},“namespace”:“psmdb”,“name”:“cron-psmdb-db-20241009020000-8t55g”,“reconcileID”:“331b0f0e-e0c6-4ffb-b35c-b6400f9a7934”,“backupCmd”:“backup [name: 2024-10-09T02:00:21Z, compression: gzip (level: default)] <ts: 0>”}

5

{“level”:“debug”,“ts”:1728439221.0823846,“msg”:“checking if backup is allowed”,“controller”:“psmdbbackup-controller”,“object”:{“name”:“cron-psmdb-db-20241009020000-8t55g”,“namespace”:“psmdb”},“namespace”:“psmdb”,“name”:“cron-psmdb-db-20241009020000-8t55g”,“reconcileID”:“230d5fdb-3b7d-4899-a2cd-f63728a128de”,“backup”:“psmdb-db”}
{“level”:“debug”,“ts”:1728439221.0824006,“msg”:“Checking for active jobs”,“controller”:“psmdbbackup-controller”,“object”:{“name”:“cron-psmdb-db-20241009020000-8t55g”,“namespace”:“psmdb”},“namespace”:“psmdb”,“name”:“cron-psmdb-db-20241009020000-8t55g”,“reconcileID”:“230d5fdb-3b7d-4899-a2cd-f63728a128de”,“currentJob”:{“Name”:“cron-psmdb-db-20241009020000-8t55g”,“Type”:0}}
{“level”:“debug”,“ts”:1728439226.1341822,“msg”:“checking if backup is allowed”,“controller”:“psmdbbackup-controller”,“object”:{“name”:“cron-psmdb-db-20241009020000-8t55g”,“namespace”:“psmdb”},“namespace”:“psmdb”,“name”:“cron-psmdb-db-20241009020000-8t55g”,“reconcileID”:“d8e21169-20f3-4148-b2e9-c46f3820ef8d”,“backup”:“psmdb-db”}
{“level”:“debug”,“ts”:1728439226.1342013,“msg”:“Checking for active jobs”,“controller”:“psmdbbackup-controller”,“object”:{“name”:“cron-psmdb-db-20241009020000-8t55g”,“namespace”:“psmdb”},“namespace”:“psmdb”,“name”:“cron-psmdb-db-20241009020000-8t55g”,“reconcileID”:“d8e21169-20f3-4148-b2e9-c46f3820ef8d”,“currentJob”:{“Name”:“cron-psmdb-db-20241009020000-8t55g”,“Type”:0}}
{“level”:“debug”,“ts”:1728442800.0358636,“msg”:“checking if backup is allowed”,“controller”:“psmdbbackup-controller”,“object”:{“name”:“cron-psmdb-db-20241009030000-6l2m2”,“namespace”:“psmdb”},“namespace”:“psmdb”,“name”:“cron-psmdb-db-20241009030000-6l2m2”,“reconcileID”:“d217fa6f-527e-4026-af65-ed920e6090de”,“backup”:“psmdb-db”}
{“level”:“debug”,“ts”:1728442800.0358863,“msg”:“Checking for active jobs”,“controller”:“psmdbbackup-controller”,“object”:{“name”:“cron-psmdb-db-20241009030000-6l2m2”,“namespace”:“psmdb”},“namespace”:“psmdb”,“name”:“cron-psmdb-db-20241009030000-6l2m2”,“reconcileID”:“d217fa6f-527e-4026-af65-ed920e6090de”,“currentJob”:{“Name”:“cron-psmdb-db-20241009030000-6l2m2”,“Type”:0}}
{“level”:“info”,“ts”:1728442810.0563312,“msg”:“Starting backup”,“controller”:“psmdbbackup-controller”,“object”:{“name”:“cron-psmdb-db-20241009030000-6l2m2”,“namespace”:“psmdb”},“namespace”:“psmdb”,“name”:“cron-psmdb-db-20241009030000-6l2m2”,“reconcileID”:“d217fa6f-527e-4026-af65-ed920e6090de”,“backup”:“cron-psmdb-db-20241009030000-6l2m2”,“storage”:“azure-blob”}
{“level”:“info”,“ts”:1728442810.0563743,“msg”:“Setting PBM config”,“controller”:“psmdbbackup-controller”,“object”:{“name”:“cron-psmdb-db-20241009030000-6l2m2”,“namespace”:“psmdb”},“namespace”:“psmdb”,“name”:“cron-psmdb-db-20241009030000-6l2m2”,“reconcileID”:“d217fa6f-527e-4026-af65-ed920e6090de”,“backup”:“psmdb-db”}
{“level”:“info”,“ts”:1728442813.84051,“msg”:“Setting pitr.oplogOnly in PBM config”,“controller”:“psmdb-controller”,“object”:{“name”:“psmdb-db”,“namespace”:“psmdb”},“namespace”:“psmdb”,“name”:“psmdb-db”,“reconcileID”:“2c969d9e-b81f-4407-add1-91d581c9c94e”,“value”:“true”}
{“level”:“info”,“ts”:1728442821.0644822,“msg”:“Sending backup command”,“controller”:“psmdbbackup-controller”,“object”:{“name”:“cron-psmdb-db-20241009030000-6l2m2”,“namespace”:“psmdb”},“namespace”:“psmdb”,“name”:“cron-psmdb-db-20241009030000-6l2m2”,“reconcileID”:“d217fa6f-527e-4026-af65-ed920e6090de”,“backupCmd”:“backup [name: 2024-10-09T03:00:21Z, compression: gzip (level: default)] <ts: 0>”}
{“level”:“debug”,“ts”:1728442821.1050568,“msg”:“checking if backup is allowed”,“controller”:“psmdbbackup-controller”,“object”:{“name”:“cron-psmdb-db-20241009030000-6l2m2”,“namespace”:“psmdb”},“namespace”:“psmdb”,“name”:“cron-psmdb-db-20241009030000-6l2m2”,“reconcileID”:“063c279b-55a3-4a8a-9c6d-22ccfd49b2fa”,“backup”:“psmdb-db”}
{“level”:“debug”,“ts”:1728442821.105073,“msg”:“Checking for active jobs”,“controller”:“psmdbbackup-controller”,“object”:{“name”:“cron-psmdb-db-20241009030000-6l2m2”,“namespace”:“psmdb”},“namespace”:“psmdb”,“name”:“cron-psmdb-db-20241009030000-6l2m2”,“reconcileID”:“063c279b-55a3-4a8a-9c6d-22ccfd49b2fa”,“currentJob”:{“Name”:“cron-psmdb-db-20241009030000-6l2m2”,“Type”:0}}
{“level”:“debug”,“ts”:1728442826.1560686,“msg”:“checking if backup is allowed”,“controller”:“psmdbbackup-controller”,“object”:{“name”:“cron-psmdb-db-20241009030000-6l2m2”,“namespace”:“psmdb”},“namespace”:“psmdb”,“name”:“cron-psmdb-db-20241009030000-6l2m2”,“reconcileID”:“de250780-589f-49e8-b5b3-734f10108436”,“backup”:“psmdb-db”}
{“level”:“debug”,“ts”:1728442826.1560884,“msg”:“Checking for active jobs”,“controller”:“psmdbbackup-controller”,“object”:{“name”:“cron-psmdb-db-20241009030000-6l2m2”,“namespace”:“psmdb”},“namespace”:“psmdb”,“name”:“cron-psmdb-db-20241009030000-6l2m2”,“reconcileID”:“de250780-589f-49e8-b5b3-734f10108436”,“currentJob”:{“Name”:“cron-psmdb-db-20241009030000-6l2m2”,“Type”:0}}
{“level”:“debug”,“ts”:1728446400.0448418,“msg”:“checking if backup is allowed”,“controller”:“psmdbbackup-controller”,“object”:{“name”:“cron-psmdb-db-20241009040000-pg9m8”,“namespace”:“psmdb”},“namespace”:“psmdb”,“name”:“cron-psmdb-db-20241009040000-pg9m8”,“reconcileID”:“e735d7eb-09d3-40b4-840f-41715b154e89”,“backup”:“psmdb-db”}
{“level”:“debug”,“ts”:1728446400.0448632,“msg”:“Checking for active jobs”,“controller”:“psmdbbackup-controller”,“object”:{“name”:“cron-psmdb-db-20241009040000-pg9m8”,“namespace”:“psmdb”},“namespace”:“psmdb”,“name”:“cron-psmdb-db-20241009040000-pg9m8”,“reconcileID”:“e735d7eb-09d3-40b4-840f-41715b154e89”,“currentJob”:{“Name”:“cron-psmdb-db-20241009040000-pg9m8”,“Type”:0}}
{“level”:“info”,“ts”:1728446410.0585768,“msg”:“Starting backup”,“controller”:“psmdbbackup-controller”,“object”:{“name”:“cron-psmdb-db-20241009040000-pg9m8”,“namespace”:“psmdb”},“namespace”:“psmdb”,“name”:“cron-psmdb-db-20241009040000-pg9m8”,“reconcileID”:“e735d7eb-09d3-40b4-840f-41715b154e89”,“backup”:“cron-psmdb-db-20241009040000-pg9m8”,“storage”:“azure-blob”}
{“level”:“info”,“ts”:1728446410.0586317,“msg”:“Setting PBM config”,“controller”:“psmdbbackup-controller”,“object”:{“name”:“cron-psmdb-db-20241009040000-pg9m8”,“namespace”:“psmdb”},“namespace”:“psmdb”,“name”:“cron-psmdb-db-20241009040000-pg9m8”,“reconcileID”:“e735d7eb-09d3-40b4-840f-41715b154e89”,“backup”:“psmdb-db”}
{“level”:“info”,“ts”:1728446414.4539754,“msg”:“Setting pitr.oplogOnly in PBM config”,“controller”:“psmdb-controller”,“object”:{“name”:“psmdb-db”,“namespace”:“psmdb”},“namespace”:“psmdb”,“name”:“psmdb-db”,“reconcileID”:“170a20dd-1e8e-45d6-9ced-e475c72efeb7”,“value”:“true”}
{“level”:“info”,“ts”:1728446421.0638158,“msg”:“Sending backup command”,“controller”:“psmdbbackup-controller”,“object”:{“name”:“cron-psmdb-db-20241009040000-pg9m8”,“namespace”:“psmdb”},“namespace”:“psmdb”,“name”:“cron-psmdb-db-20241009040000-pg9m8”,“reconcileID”:“e735d7eb-09d3-40b4-840f-41715b154e89”,“backupCmd”:“backup [name: 2024-10-09T04:00:21Z, compression: gzip (level: default)] <ts: 0>”}
{“level”:“debug”,“ts”:1728446421.0962088,“msg”:“checking if backup is allowed”,“controller”:“psmdbbackup-controller”,“object”:{“name”:“cron-psmdb-db-20241009040000-pg9m8”,“namespace”:“psmdb”},“namespace”:“psmdb”,“name”:“cron-psmdb-db-20241009040000-pg9m8”,“reconcileID”:“4cc5a67d-5e2e-44bc-90d6-11e8bee6bfb1”,“backup”:“psmdb-db”}
{“level”:“debug”,“ts”:1728446421.096227,“msg”:“Checking for active jobs”,“controller”:“psmdbbackup-controller”,“object”:{“name”:“cron-psmdb-db-20241009040000-pg9m8”,“namespace”:“psmdb”},“namespace”:“psmdb”,“name”:“cron-psmdb-db-20241009040000-pg9m8”,“reconcileID”:“4cc5a67d-5e2e-44bc-90d6-11e8bee6bfb1”,“currentJob”:{“Name”:“cron-psmdb-db-20241009040000-pg9m8”,“Type”:0}}
{“level”:“debug”,“ts”:1728446426.159586,“msg”:“checking if backup is allowed”,“controller”:“psmdbbackup-controller”,“object”:{“name”:“cron-psmdb-db-20241009040000-pg9m8”,“namespace”:“psmdb”},“namespace”:“psmdb”,“name”:“cron-psmdb-db-20241009040000-pg9m8”,“reconcileID”:“230ff64a-54ad-4d8d-9bb2-0cec899be1b7”,“backup”:“psmdb-db”}
{“level”:“debug”,“ts”:1728446426.1596053,“msg”:“Checking for active jobs”,“controller”:“psmdbbackup-controller”,“object”:{“name”:“cron-psmdb-db-20241009040000-pg9m8”,“namespace”:“psmdb”},“namespace”:“psmdb”,“name”:“cron-psmdb-db-20241009040000-pg9m8”,“reconcileID”:“230ff64a-54ad-4d8d-9bb2-0cec899be1b7”,“currentJob”:{“Name”:“cron-psmdb-db-20241009040000-pg9m8”,“Type”:0}}
{“level”:“debug”,“ts”:1728450000.0400693,“msg”:“checking if backup is allowed”,“controller”:“psmdbbackup-controller”,“object”:{“name”:“cron-psmdb-db-20241009050000-w6gpm”,“namespace”:“psmdb”},“namespace”:“psmdb”,“name”:“cron-psmdb-db-20241009050000-w6gpm”,“reconcileID”:“8ab94bb0-ef99-4ff9-80e0-c6c634084460”,“backup”:“psmdb-db”}
{“level”:“debug”,“ts”:1728450000.0400887,“msg”:“Checking for active jobs”,“controller”:“psmdbbackup-controller”,“object”:{“name”:“cron-psmdb-db-20241009050000-w6gpm”,“namespace”:“psmdb”},“namespace”:“psmdb”,“name”:“cron-psmdb-db-20241009050000-w6gpm”,“reconcileID”:“8ab94bb0-ef99-4ff9-80e0-c6c634084460”,“currentJob”:{“Name”:“cron-psmdb-db-20241009050000-w6gpm”,“Type”:0}}
{“level”:“info”,“ts”:1728450010.0714107,“msg”:“Starting backup”,“controller”:“psmdbbackup-controller”,“object”:{“name”:“cron-psmdb-db-20241009050000-w6gpm”,“namespace”:“psmdb”},“namespace”:“psmdb”,“name”:“cron-psmdb-db-20241009050000-w6gpm”,“reconcileID”:“8ab94bb0-ef99-4ff9-80e0-c6c634084460”,“backup”:“cron-psmdb-db-20241009050000-w6gpm”,“storage”:“azure-blob”}
{“level”:“info”,“ts”:1728450010.0714602,“msg”:“Setting PBM config”,“controller”:“psmdbbackup-controller”,“object”:{“name”:“cron-psmdb-db-20241009050000-w6gpm”,“namespace”:“psmdb”},“namespace”:“psmdb”,“name”:“cron-psmdb-db-20241009050000-w6gpm”,“reconcileID”:“8ab94bb0-ef99-4ff9-80e0-c6c634084460”,“backup”:“psmdb-db”}
{“level”:“info”,“ts”:1728450010.157176,“msg”:“Setting pitr.oplogOnly in PBM config”,“controller”:“psmdb-controller”,“object”:{“name”:“psmdb-db”,“namespace”:“psmdb”},“namespace”:“psmdb”,“name”:“psmdb-db”,“reconcileID”:“b4c51b4d-9f8f-437c-ac14-4da24d0a45db”,“value”:“true”}
{“level”:“info”,“ts”:1728450021.076537,“msg”:“Sending backup command”,“controller”:“psmdbbackup-controller”,“object”:{“name”:“cron-psmdb-db-20241009050000-w6gpm”,“namespace”:“psmdb”},“namespace”:“psmdb”,“name”:“cron-psmdb-db-20241009050000-w6gpm”,“reconcileID”:“8ab94bb0-ef99-4ff9-80e0-c6c634084460”,“backupCmd”:“backup [name: 2024-10-09T05:00:21Z, compression: gzip (level: default)] <ts: 0>”}

6

I don’t see any attempts to delete or remove files. I don’t see any errors either. Can you try to force delete a backup?

7

Force delete works fine I just run kubectl delete psmdb-backup and it got deleted successfully from Azure blob storage container. Are there any differences between kubectl delete and automatic delete from a code perspective?

8

Hey.

We had a call with Mozuffer and also discussed it in the team.
We have confirmed that in the latest version of the operator we have this issue with backup rotation.
It is tracked here: Issue navigator - Percona JIRA

This thread also relates to Older backups are not getting deleted, with the keep argument specified