Description:
I’ve setup two mongo clusters on different environments using the psmdb-db and psmdb-operator helm charts. I’ve setup for both of those cluster the following configuration concerning backups:
backup:
enabled: true
storages:
s3-eu-west-3:
type: s3
s3:
bucket: xxxxxxxxxxx
region: eu-west-3
credentialsSecret: s3-credentials
tasks:
- name: hourly
enabled: true
schedule: "0 * * * *"
keep: 24
storageName: s3-eu-west-3
compressionType: gzip
- name: daily
enabled: true
schedule: "0 0 * * *"
keep: 7
storageName: s3-eu-west-3
compressionType: gzip
- name: weekly
enabled: true
schedule: "0 0 * * 0"
keep: 5
storageName: s3-eu-west-3
compressionType: gzip
- name: monthly
enabled: true
schedule: "0 0 1 * *"
keep: 6
storageName: s3-eu-west-3
compressionType: gzip
My backups are working fine but they are not getting deleted which cause lots of usage on my aws s3. Plus, I am not sure if all tasks are being executed or only hourly, since they are not named differently. For instance, on my s3, I have hourly backups dating from almost a year! I checked my logs (will add them), and it never states that it will delete or remove anything which is weird. On one of my environment, I tried executing the pbm cleanup command and it did well so it doesn’t seem to have to do with permissions.
Steps to Reproduce:
Create a mongodb cluster using the previously stated backup configuration
Version:
Percona psmdb-operator and psmdb-db charts 1.17.0
Logs:
operator:
2024-10-17T10:00:27.567Z INFO balancer enabled {"controller": "psmdb-controller", "object": {"name":"mongodb-psmdb-db","namespace":"mongodb"}, "namespace": "mongodb", "name": "mongodb-psmdb-db", "reconcileID": "14bfbd08-82ce-4172-bad4-d1a07a2db384"}
2024-10-17T10:00:21.267Z INFO Sending backup command {"controller": "psmdbbackup-controller", "object": {"name":"cron-mongodb-psmdb-db-20241017100000-gtfr7","namespace":"mongodb"}, "namespace": "mongodb", "name": "cron-mongodb-psmdb-db-20241017100000-gtfr7", "reconcileID": "3fb2453d-75f5-4c45-8042-1ae4b84bcc43", "backupCmd": "backup [name: 2024-10-17T10:00:21Z, compression: gzip (level: default)] <ts: 0>"}
2024-10-17T10:00:10.244Z INFO Setting PBM config {"controller": "psmdbbackup-controller", "object": {"name":"cron-mongodb-psmdb-db-20241017100000-gtfr7","namespace":"mongodb"}, "namespace": "mongodb", "name": "cron-mongodb-psmdb-db-20241017100000-gtfr7", "reconcileID": "3fb2453d-75f5-4c45-8042-1ae4b84bcc43", "backup": "mongodb-psmdb-db"}
2024-10-17T10:00:10.244Z INFO Starting backup {"controller": "psmdbbackup-controller", "object": {"name":"cron-mongodb-psmdb-db-20241017100000-gtfr7","namespace":"mongodb"}, "namespace": "mongodb", "name": "cron-mongodb-psmdb-db-20241017100000-gtfr7", "reconcileID": "3fb2453d-75f5-4c45-8042-1ae4b84bcc43", "backup": "cron-mongodb-psmdb-db-20241017100000-gtfr7", "storage": "s3-eu-west-3"}
2024-10-17T09:05:58.727Z INFO add new job {"controller": "psmdb-controller", "object": {"name":"mongodb-psmdb-db","namespace":"mongodb"}, "namespace": "mongodb", "name": "mongodb-psmdb-db", "reconcileID": "f0d7662f-310b-4ee7-bb9b-fc435e90ff79", "name": "ensure-version/mongodb/mongodb-psmdb-db", "schedule": "0 2 * * *"}
2024-10-17T09:05:54.381Z INFO Creating or updating backup job {"controller": "psmdb-controller", "object": {"name":"mongodb-psmdb-db","namespace":"mongodb"}, "namespace": "mongodb", "name": "mongodb-psmdb-db", "reconcileID": "f0d7662f-310b-4ee7-bb9b-fc435e90ff79", "name": "monthly", "namespace": "mongodb", "schedule": "0 0 1 * *"}
2024-10-17T09:05:54.381Z INFO Creating or updating backup job {"controller": "psmdb-controller", "object": {"name":"mongodb-psmdb-db","namespace":"mongodb"}, "namespace": "mongodb", "name": "mongodb-psmdb-db", "reconcileID": "f0d7662f-310b-4ee7-bb9b-fc435e90ff79", "name": "weekly", "namespace": "mongodb", "schedule": "0 0 * * 0"}
2024-10-17T09:05:54.381Z INFO Creating or updating backup job {"controller": "psmdb-controller", "object": {"name":"mongodb-psmdb-db","namespace":"mongodb"}, "namespace": "mongodb", "name": "mongodb-psmdb-db", "reconcileID": "f0d7662f-310b-4ee7-bb9b-fc435e90ff79", "name": "daily", "namespace": "mongodb", "schedule": "0 0 * * *"}
2024-10-17T09:05:54.381Z INFO Creating or updating backup job {"controller": "psmdb-controller", "object": {"name":"mongodb-psmdb-db","namespace":"mongodb"}, "namespace": "mongodb", "name": "mongodb-psmdb-db", "reconcileID": "f0d7662f-310b-4ee7-bb9b-fc435e90ff79", "name": "hourly", "namespace": "mongodb", "schedule": "0 * * * *"}
2024-10-17T09:05:54.261Z INFO Starting workers {"controller": "psmdbrestore-controller", "worker count": 1}
2024-10-17T09:05:54.261Z INFO Starting workers {"controller": "psmdbbackup-controller", "worker count": 1}
2024-10-17T09:05:54.251Z INFO Starting workers {"controller": "psmdb-controller", "worker count": 1}
2024-10-17T09:05:53.794Z INFO Starting Controller {"controller": "psmdbbackup-controller"}
2024-10-17T09:05:53.794Z INFO Starting EventSource {"controller": "psmdbbackup-controller", "source": "kind source: *v1.Pod"}
2024-10-17T09:05:53.794Z INFO Starting EventSource {"controller": "psmdbbackup-controller", "source": "kind source: *v1.PerconaServerMongoDBBackup"}
2024-10-17T09:05:53.794Z INFO Starting Controller {"controller": "psmdb-controller"}
2024-10-17T09:05:53.793Z INFO Starting Controller {"controller": "psmdbrestore-controller"}
2024-10-17T09:05:53.793Z INFO Starting EventSource {"controller": "psmdbrestore-controller", "source": "kind source: *v1.Pod"}
2024-10-17T09:05:53.793Z INFO Starting EventSource {"controller": "psmdbrestore-controller", "source": "kind source: *v1.PerconaServerMongoDBRestore"}
2024-10-17T09:05:53.793Z INFO Starting EventSource {"controller": "psmdb-controller", "source": "kind source: *v1.PerconaServerMongoDB"}
I1017 09:05:53.793075 1 leaderelection.go:260] successfully acquired lease mongodb/08db0feb.percona.com
I1017 09:05:38.594356 1 leaderelection.go:250] attempting to acquire leader lease mongodb/08db0feb.percona.com...
2024-10-17T09:05:38.594Z INFO starting server {"name": "health probe", "addr": "[::]:8081"}
2024-10-17T09:05:38.593Z INFO controller-runtime.metrics Serving metrics server {"bindAddress": ":8080", "secure": false}
2024-10-17T09:05:38.593Z INFO controller-runtime.metrics Starting metrics server
2024-10-17T09:05:38.581Z INFO server version {"platform": "kubernetes", "version": "v1.30.4-eks-a737599"}
2024-10-17T09:05:38.551Z INFO setup Manager starting up {"gitCommit": "5019408f1fe40483fc5effaf61ab3f672765b189", "gitBranch": "release-1-17-0", "goVersion": "go1.22.6", "os": "linux", "arch": "amd64"}
Expected Result:
Deletion of older backups
Actual Result:
Older backups are still there
Additional Information:
If you need any additional info please feel free to ask, I will provide everyting needed.
It seems to be linked to this issue