# Backup retention not applied in v1.17.0

**URL:** <https://forums.percona.com/t/backup-retention-not-applied-in-v1-17-0/38904>\
**Category:** Percona Operator for MongoDB\
**Created:** [July 31, 2025, 1:34pm UTC](https://forums.percona.com/t/backup-retention-not-applied-in-v1-17-0/38904 "2025-07-31T13:34:41Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![SuperHammer](https://avatars.discourse-cdn.com/v4/letter/s/ccd318/32.png) [@SuperHammer](https://forums.percona.com/u/SuperHammer)\
**Post date:** [July 31, 2025, 1:34pm UTC](https://forums.percona.com/t/backup-retention-not-applied-in-v1-17-0/38904/1 "2025-07-31T13:34:41Z")

</div>

## Description:

In an OKD SNO environment, installing the operator and running it works correctly. When making backups to a local minio, backups work fine, but the set retention is not applied, resulting in a large number of old backups.

## Steps to Reproduce:

- Install an OKD SNO cluster
- Through the OperatorHub section of the web console, install “Percona Distribution for MongoDB Operator” 1.17.0 provided by Percona
- Follow the installation steps and use the following CRD when required:

```yaml
apiVersion: psmdb.percona.com/v1
kind: PerconaServerMongoDB
metadata:
  name: prod
  finalizers:
    - percona.com/delete-psmdb-pods-in-order
spec:
  replsets:
    - expose:
        enabled: false
        exposeType: ClusterIP
      resources:
        limits:
          cpu: 300m
          memory: 0.5G
        requests:
          cpu: 300m
          memory: 0.5G
      affinity:
        antiAffinityTopologyKey: none
      arbiter:
        affinity:
          antiAffinityTopologyKey: none
        enabled: false
        size: 0
      nonvoting:
        resources:
          limits:
            cpu: 300m
            memory: 0.5G
          requests:
            cpu: 300m
            memory: 0.5G
        affinity:
          antiAffinityTopologyKey: none
        volumeSpec:
          persistentVolumeClaim:
            resources:
              requests:
                storage: 3Gi
        podDisruptionBudget: {}
        enabled: false
        size: 1
      volumeSpec:
        persistentVolumeClaim:
          resources:
            requests:
              storage: 5Gi
      podDisruptionBudget: {}
      name: rs0
      size: 3
  multiCluster:
    enabled: false
    DNSSuffix: svc.clusterset.local
  upgradeOptions:
    versionServiceEndpoint: 'https://check.percona.com'
    apply: disabled
    schedule: 0 2 * * *
    setFCV: false
  backup:
    pitr:
      enabled: false
      compressionType: gzip
      compressionLevel: 6
    enabled: true
    image: percona/percona-backup-mongodb:2.5.0
    serviceAccountName: percona-server-mongodb-operator
    storages:
      minio:
        type: s3
        s3:
          bucket: pmdb-prod-backup
          region: us-east-1
          credentialsSecret: prod-backup-minio-s3
          endpointUrl: http://minio-api.minio.svc.cluster.local:9000
          insecureSkipTLSVerify: false
    tasks:
      - compressionType: gzip
        enabled: true
        keep: 3
        name: daily-backup
        schedule: 0 0 * * *
        storageName: minio
  secrets:
    users: pmdb-mongodb-prod
    encryptionKey: my-cluster-name-mongodb-encryption-key
  sharding:
    configsvrReplSet:
      expose:
        enabled: false
        exposeType: ClusterIP
      resources:
        limits:
          cpu: 300m
          memory: 0.5G
        requests:
          cpu: 300m
          memory: 0.5G
      affinity:
        antiAffinityTopologyKey: none
      volumeSpec:
        persistentVolumeClaim:
          resources:
            requests:
              storage: 3Gi
      podDisruptionBudget: {}
      size: 1
    mongos:
      expose:
        exposeType: ClusterIP
      resources:
        limits:
          cpu: 300m
          memory: 0.5G
        requests:
          cpu: 300m
          memory: 0.5G
      affinity:
        antiAffinityTopologyKey: none
      podDisruptionBudget: {}
      size: 1
    enabled: false
  pmm:
    enabled: false
    image: percona/pmm-client:2.42.0
    serverHost: monitoring-service
  crVersion: 1.17.0
  image: percona/percona-server-mongodb:7.0.12-7
  imagePullPolicy: Always
  allowUnsafeConfigurations: false
  updateStrategy: SmartUpdate

```

- Everything works correctly, the backups are created on a local Minio also installed in the same cluster with a basic configuration
- Backups retention is not applied after 3 days (.spec.backup.tasks[0].keep)

## Version:

Operator version: percona-server-mongodb-operator.v1.17.0 (installed through OperatorHub)

OKD 4.15 version 4.15.0-0.okd-2024-03-10-010116 with kubernetes v1.28.2-3598+6e2789bbd58938-dirty

## Logs:

The operator pod has many restarts:

```auto
# oc -n prod-pmdb get pods
NAME READY STATUS RESTARTS AGE
percona-server-mongodb-operator-65cd6cfc48-ktwds 1/1 Running 373 (14h ago) 258d
prod-rs0-0 2/2 Running 3 (65d ago) 258d
prod-rs0-1 2/2 Running 2 258d
prod-rs0-2 2/2 Running 3 (84d ago) 258d

```

The current logs keep repeating the same messages:

```auto
<I'll post the logs in a separate message or otherwise the forum won't let me post>

```

The pod on restart fails with exit code 2:

```auto
oc -n prod-pmdb describe pod percona-server-mongodb-operator-65cd6cfc48-ktwds
...
    Command:
      percona-server-mongodb-operator
    State: Running
      Started: Wed, 30 Jul 2025 18:07:47 -0400
    Last State: Terminated
      Reason: Error
      Exit Code: 2
      Started: Tue, 29 Jul 2025 10:59:52 -0400
      Finished: Wed, 30 Jul 2025 18:07:45 -0400
    Ready: True
    Restart Count: 373
...

```

These are the last logs for when it was restarted:

```auto
<I'll post the logs in a separate message or otherwise the forum won't let me post>

```

## Expected Result:

The retention should be applied and only the last 3 days of backups should be kept.

## Actual Result:

The retention is not applied and the backups are not deleted.

## Additional Information:

The cluster is running on a single node (OKD SNO), but it shouldn’t matter from an operator perspective.

The CR has a status of “Ready”:

```auto
# oc -n prod-pmdb get PerconaServerMongoDB prod
NAME ENDPOINT STATUS AGE
prod prod-rs0.prod-pmdb.svc.cluster.local ready 258d

```

Full CR with describe:

```auto
# oc -n prod-pmdb describe PerconaServerMongoDB prod
Name: prod
Namespace: prod-pmdb
Labels: <none>
Annotations: <none>
API Version: psmdb.percona.com/v1
Kind: PerconaServerMongoDB
Metadata:
  Creation Timestamp: 2024-11-15T09:06:09Z
  Finalizers:
    percona.com/delete-psmdb-pods-in-order
  Generation: 1
  Resource Version: 95156491
  UID: ab68cd17-20b3-4e3e-88af-7c12077eb0f3
Spec:
  Allow Unsafe Configurations: false
  Backup:
    Enabled: true
    Image: percona/percona-backup-mongodb:2.5.0
    Pitr:
      Compression Level: 6
      Compression Type: gzip
      Enabled: false
    Service Account Name: percona-server-mongodb-operator
    Storages:
      Minio:
        s3:
          Bucket: pmdb-prod-backup
          Credentials Secret: prod-backup-minio-s3
          Endpoint URL: http://minio-api.minio.svc.cluster.local:9000
          Insecure Skip TLS Verify: false
          Region: us-east-1
        Type: s3
    Tasks:
      Compression Type: gzip
      Enabled: true
      Keep: 3
      Name: daily-backup
      Schedule: 0 0 * * *
      Storage Name: minio
  Cr Version: 1.17.0
  Image: percona/percona-server-mongodb:7.0.12-7
  Image Pull Policy: Always
  Multi Cluster:
    DNS Suffix: svc.clusterset.local
    Enabled: false
  Pmm:
    Enabled: false
    Image: percona/pmm-client:2.42.0
    Server Host: monitoring-service
  Replsets:
    Affinity:
      Anti Affinity Topology Key: none
    Arbiter:
      Affinity:
        Anti Affinity Topology Key: none
      Enabled: false
      Size: 0
    Expose:
      Enabled: false
      Expose Type: ClusterIP
    Name: rs0
    Nonvoting:
      Affinity:
        Anti Affinity Topology Key: none
      Enabled: false
      Pod Disruption Budget:
      Resources:
        Limits:
          Cpu: 300m
          Memory: 0.5G
        Requests:
          Cpu: 300m
          Memory: 0.5G
      Size: 1
      Volume Spec:
        Persistent Volume Claim:
          Resources:
            Requests:
              Storage: 3Gi
    Pod Disruption Budget:
    Resources:
      Limits:
        Cpu: 300m
        Memory: 0.5G
      Requests:
        Cpu: 300m
        Memory: 0.5G
    Size: 3
    Volume Spec:
      Persistent Volume Claim:
        Resources:
          Requests:
            Storage: 5Gi
  Secrets:
    Encryption Key: my-cluster-name-mongodb-encryption-key
    Users: pmdb-mongodb-prod
  Sharding:
    Configsvr Repl Set:
      Affinity:
        Anti Affinity Topology Key: none
      Expose:
        Enabled: false
        Expose Type: ClusterIP
      Pod Disruption Budget:
      Resources:
        Limits:
          Cpu: 300m
          Memory: 0.5G
        Requests:
          Cpu: 300m
          Memory: 0.5G
      Size: 1
      Volume Spec:
        Persistent Volume Claim:
          Resources:
            Requests:
              Storage: 3Gi
    Enabled: false
    Mongos:
      Affinity:
        Anti Affinity Topology Key: none
      Expose:
        Expose Type: ClusterIP
      Pod Disruption Budget:
      Resources:
        Limits:
          Cpu: 300m
          Memory: 0.5G
        Requests:
          Cpu: 300m
          Memory: 0.5G
      Size: 1
  Update Strategy: SmartUpdate
  Upgrade Options:
    Apply: disabled
    Schedule: 0 2 * * *
    Set FCV: false
    Version Service Endpoint: https://check.percona.com
Status:
  Conditions:
    Last Transition Time: 2024-11-15T09:06:11Z
    Status: True
    Type: initializing
    Last Transition Time: 2024-11-15T09:07:12Z
    Message: rs0: ready
    Reason: RSReady
    Status: True
    Type: ready
    Last Transition Time: 2024-11-15T09:07:12Z
    Status: True
    Type: initializing
    Last Transition Time: 2024-11-15T09:07:23Z
    Status: True
    Type: ready
    Last Transition Time: 2025-05-08T00:00:52Z
    Status: True
    Type: initializing
    Last Transition Time: 2025-05-08T00:01:23Z
    Message: rs0: ready
    Reason: RSReady
    Status: True
    Type: ready
    Last Transition Time: 2025-05-08T00:02:36Z
    Status: True
    Type: initializing
    Last Transition Time: 2025-05-08T00:02:47Z
    Message: rs0: ready
    Reason: RSReady
    Status: True
    Type: ready
    Last Transition Time: 2025-05-08T00:02:47Z
    Status: True
    Type: initializing
    Last Transition Time: 2025-05-08T00:02:47Z
    Status: True
    Type: ready
    Last Transition Time: 2025-05-27T00:07:34Z
    Status: True
    Type: initializing
    Last Transition Time: 2025-05-27T00:07:50Z
    Message: rs0: ready
    Reason: RSReady
    Status: True
    Type: ready
    Last Transition Time: 2025-07-31T13:15:50Z
    Status: True
    Type: initializing
    Last Transition Time: 2025-07-31T13:16:06Z
    Message: rs0: ready
    Reason: RSReady
    Status: True
    Type: ready
    Last Transition Time: 2025-07-31T13:16:06Z
    Status: True
    Type: initializing
    Last Transition Time: 2025-07-31T13:16:11Z
    Status: True
    Type: ready
  Host: prod-rs0.prod-pmdb.svc.cluster.local
  Mongo Image: percona/percona-server-mongodb:7.0.12-7
  Mongo Version: 7.0.12-7
  Observed Generation: 1
  Ready: 3
  Replsets:
    rs0:
      Initialized: true
      Ready: 3
      Size: 3
      Status: ready
  Size: 3
  State: ready
Events: <none>

```

The backups are really small as the total database size is \<1GB on both disk and DB.

As of now there are more than 200+ backups ready but not deleted:

```auto
<I'll post the output in a separate message or otherwise the forum won't let me post>

```

Here is the full YAML of the oldest backup:

```auto
# oc -n prod-pmdb get psmdb-backup cron-prod-20241116000000-wcj8f
NAME CLUSTER STORAGE DESTINATION TYPE STATUS COMPLETED AGE
cron-prod-20241116000000-wcj8f prod minio s3://pmdb-prod-backup/2024-11-16T00:00:21Z logical ready 257d 257d
# oc -n prod-pmdb get psmdb-backup cron-prod-20241116000000-wcj8f -o yaml
apiVersion: psmdb.percona.com/v1
kind: PerconaServerMongoDBBackup
metadata:
  creationTimestamp: "2024-11-16T00:00:00Z"
  finalizers:
  - percona.com/delete-backup
  generateName: cron-prod-20241116000000-
  generation: 1
  labels:
    app.kubernetes.io/instance: prod
    app.kubernetes.io/managed-by: percona-server-mongodb-operator
    app.kubernetes.io/name: percona-server-mongodb
    app.kubernetes.io/part-of: percona-server-mongodb
    percona.com/backup-ancestor: daily-backup
    percona.com/backup-type: cron
    percona.com/cluster: prod
  name: cron-prod-20241116000000-wcj8f
  namespace: prod-pmdb
  resourceVersion: "2570408"
  uid: 6f2f9b3b-8a55-43a4-b39b-f5fc6f7b2b2e
spec:
  clusterName: prod
  compressionType: gzip
  storageName: minio
  type: logical
status:
  completed: "2024-11-16T00:00:31Z"
  destination: s3://pmdb-prod-backup/2024-11-16T00:00:21Z
  lastTransition: "2024-11-16T00:00:31Z"
  pbmName: "2024-11-16T00:00:21Z"
  pbmPod: prod-rs0-0
  pbmPods:
    rs0: prod-rs0-0
  replsetNames:
  - rs0
  s3:
    bucket: pmdb-prod-backup
    credentialsSecret: prod-backup-minio-s3
    endpointUrl: http://minio-api.minio.svc.cluster.local:9000
    region: us-east-1
    serverSideEncryption: {}
  start: "2024-11-16T00:00:21Z"
  state: ready
  storageName: minio
  type: logical

```

And if I try to delete it, it works correctly:

```auto
# oc -n prod-pmdb delete psmdb-backup cron-prod-20241116000000-wcj8f
^C
# oc -n prod-pmdb delete psmdb-backup cron-prod-20241116000000-wcj8f
Error from server (NotFound): perconaservermongodbbackups.psmdb.percona.com "cron-prod-20241116000000-wcj8f" not found

```

As soon as possible I’d like to replicate this behaviour in a specific lab, but I’m opening an issue in the meantime in case it could be of help and in case anybody could suggest any more specific troubleshooting steps.

---

<div class="post-metadata">

**Author:** ![SuperHammer](https://avatars.discourse-cdn.com/v4/letter/s/ccd318/32.png) [@SuperHammer](https://forums.percona.com/u/SuperHammer)\
**Post date:** [July 31, 2025, 1:38pm UTC](https://forums.percona.com/t/backup-retention-not-applied-in-v1-17-0/38904/2 "2025-07-31T13:38:53Z")

</div>

Backup CRs:

```auto
oc -n prod-pmdb get psmdb-backup
NAME CLUSTER STORAGE DESTINATION TYPE STATUS COMPLETED AGE
...
cron-prod-20241227000000-h4g5j prod minio s3://pmdb-prod-backup/2024-12-27T00:00:21Z logical ready 216d 216d
cron-prod-20241228000000-9956b prod minio s3://pmdb-prod-backup/2024-12-28T00:00:21Z logical ready 215d 215d
cron-prod-20241229000000-7nxqr prod minio s3://pmdb-prod-backup/2024-12-29T00:00:21Z logical ready 214d 214d
cron-prod-20241230000000-m5ck9 prod minio s3://pmdb-prod-backup/2024-12-30T00:00:21Z logical ready 213d 213d
cron-prod-20241231000000-xzp8n prod minio s3://pmdb-prod-backup/2024-12-31T00:00:21Z logical ready 212d 212d
cron-prod-20250101000000-ldkpn prod minio s3://pmdb-prod-backup/2025-01-01T00:00:21Z logical ready 211d 211d
cron-prod-20250102000000-5jmdf prod minio s3://pmdb-prod-backup/2025-01-02T00:00:21Z logical ready 210d 210d
cron-prod-20250103000000-5s8c6 prod minio s3://pmdb-prod-backup/2025-01-03T00:00:21Z logical ready 209d 209d
cron-prod-20250104000000-2x2pj prod minio s3://pmdb-prod-backup/2025-01-04T00:00:21Z logical ready 208d 208d
cron-prod-20250105000000-k4f2v prod minio s3://pmdb-prod-backup/2025-01-05T00:00:21Z logical ready 207d 207d
cron-prod-20250106000000-slcqr prod minio s3://pmdb-prod-backup/2025-01-06T00:00:21Z logical ready 206d 206d
cron-prod-20250107000000-fvd2m prod minio s3://pmdb-prod-backup/2025-01-07T00:00:21Z logical ready 205d 205d
cron-prod-20250108000000-vpfdk prod minio s3://pmdb-prod-backup/2025-01-08T00:00:21Z logical ready 204d 204d
cron-prod-20250109000000-ff9rf prod minio s3://pmdb-prod-backup/2025-01-09T00:00:21Z logical ready 203d 203d
cron-prod-20250110000000-swqrl prod minio s3://pmdb-prod-backup/2025-01-10T00:00:21Z logical ready 202d 202d
cron-prod-20250111000000-n2lw6 prod minio s3://pmdb-prod-backup/2025-01-11T00:00:21Z logical ready 201d 201d
cron-prod-20250112000000-g2bdf prod minio s3://pmdb-prod-backup/2025-01-12T00:00:21Z logical ready 200d 200d
cron-prod-20250113000000-7r42l prod minio s3://pmdb-prod-backup/2025-01-13T00:00:21Z logical ready 199d 199d
cron-prod-20250114000000-nmrh4 prod minio s3://pmdb-prod-backup/2025-01-14T00:00:21Z logical ready 198d 198d
cron-prod-20250115000000-shn7m prod minio s3://pmdb-prod-backup/2025-01-15T00:00:21Z logical ready 197d 197d
cron-prod-20250116000000-zzxtv prod minio s3://pmdb-prod-backup/2025-01-16T00:00:21Z logical ready 196d 196d
cron-prod-20250117000000-s5p5b prod minio s3://pmdb-prod-backup/2025-01-17T00:00:21Z logical ready 195d 195d
cron-prod-20250118000000-tn82q prod minio s3://pmdb-prod-backup/2025-01-18T00:00:21Z logical ready 194d 194d
cron-prod-20250119000000-ns5sn prod minio s3://pmdb-prod-backup/2025-01-19T00:00:21Z logical ready 193d 193d
cron-prod-20250120000000-8wcm6 prod minio s3://pmdb-prod-backup/2025-01-20T00:00:21Z logical ready 192d 192d
cron-prod-20250121000000-jw9mx prod minio s3://pmdb-prod-backup/2025-01-21T00:00:21Z logical ready 191d 191d
cron-prod-20250122000000-w6ms7 prod minio s3://pmdb-prod-backup/2025-01-22T00:00:21Z logical ready 190d 190d
cron-prod-20250123000000-5xn9w prod minio s3://pmdb-prod-backup/2025-01-23T00:00:21Z logical ready 189d 189d
cron-prod-20250124000000-cl2jg prod minio s3://pmdb-prod-backup/2025-01-24T00:00:21Z logical ready 188d 188d
cron-prod-20250125000000-kndjp prod minio s3://pmdb-prod-backup/2025-01-25T00:00:21Z logical ready 187d 187d
cron-prod-20250126000000-ntv87 prod minio s3://pmdb-prod-backup/2025-01-26T00:00:21Z logical ready 186d 186d
cron-prod-20250127000000-27frf prod minio s3://pmdb-prod-backup/2025-01-27T00:00:21Z logical ready 185d 185d
cron-prod-20250128000000-8ghcf prod minio s3://pmdb-prod-backup/2025-01-28T00:00:21Z logical ready 184d 184d
cron-prod-20250129000000-p5rns prod minio s3://pmdb-prod-backup/2025-01-29T00:00:21Z logical ready 183d 183d
cron-prod-20250130000000-fsxvk prod minio s3://pmdb-prod-backup/2025-01-30T00:00:21Z logical ready 182d 182d
cron-prod-20250131000000-cspb5 prod minio s3://pmdb-prod-backup/2025-01-31T00:00:21Z logical ready 181d 181d
cron-prod-20250201000000-rgz4d prod minio s3://pmdb-prod-backup/2025-02-01T00:00:21Z logical ready 180d 180d
cron-prod-20250202000000-wjwlx prod minio s3://pmdb-prod-backup/2025-02-02T00:00:21Z logical ready 179d 179d
cron-prod-20250203000000-xhk9h prod minio s3://pmdb-prod-backup/2025-02-03T00:00:21Z logical ready 178d 178d
cron-prod-20250204000000-wssnw prod minio s3://pmdb-prod-backup/2025-02-04T00:00:21Z logical ready 177d 177d
cron-prod-20250205000000-82hcn prod minio s3://pmdb-prod-backup/2025-02-05T00:00:21Z logical ready 176d 176d
cron-prod-20250206000000-85dqc prod minio s3://pmdb-prod-backup/2025-02-06T00:00:21Z logical ready 175d 175d
cron-prod-20250207000000-m58tx prod minio s3://pmdb-prod-backup/2025-02-07T00:00:21Z logical ready 174d 174d
cron-prod-20250208000000-gntcl prod minio s3://pmdb-prod-backup/2025-02-08T00:00:21Z logical ready 173d 173d
cron-prod-20250209000000-7gwlr prod minio s3://pmdb-prod-backup/2025-02-09T00:00:21Z logical ready 172d 172d
cron-prod-20250210000000-5z4qk prod minio s3://pmdb-prod-backup/2025-02-10T00:00:21Z logical ready 171d 171d
cron-prod-20250211000000-2xz2g prod minio s3://pmdb-prod-backup/2025-02-11T00:00:21Z logical ready 170d 170d
cron-prod-20250212000000-dmhwv prod minio s3://pmdb-prod-backup/2025-02-12T00:00:21Z logical ready 169d 169d
cron-prod-20250213000000-frtfr prod minio s3://pmdb-prod-backup/2025-02-13T00:00:21Z logical ready 168d 168d
cron-prod-20250214000000-fxjf4 prod minio s3://pmdb-prod-backup/2025-02-14T00:00:21Z logical ready 167d 167d
cron-prod-20250215000000-v7pvp prod minio s3://pmdb-prod-backup/2025-02-15T00:00:21Z logical ready 166d 166d
cron-prod-20250216000000-6xbf4 prod minio s3://pmdb-prod-backup/2025-02-16T00:00:21Z logical ready 165d 165d
cron-prod-20250217000000-mvzxl prod minio s3://pmdb-prod-backup/2025-02-17T00:00:21Z logical ready 164d 164d
cron-prod-20250218000000-8pdgx prod minio s3://pmdb-prod-backup/2025-02-18T00:00:21Z logical ready 163d 163d
cron-prod-20250219000000-8w4s6 prod minio s3://pmdb-prod-backup/2025-02-19T00:00:21Z logical ready 162d 162d
cron-prod-20250220000000-blxjx prod minio s3://pmdb-prod-backup/2025-02-20T00:00:21Z logical ready 161d 161d
cron-prod-20250221000000-lrhkn prod minio s3://pmdb-prod-backup/2025-02-21T00:00:21Z logical ready 160d 160d
cron-prod-20250222000000-kfwwh prod minio s3://pmdb-prod-backup/2025-02-22T00:00:21Z logical ready 159d 159d
cron-prod-20250223000000-lcx5w prod minio s3://pmdb-prod-backup/2025-02-23T00:00:21Z logical ready 158d 158d
cron-prod-20250224000000-jjxl7 prod minio s3://pmdb-prod-backup/2025-02-24T00:00:21Z logical ready 157d 157d
cron-prod-20250225000000-qqhw5 prod minio s3://pmdb-prod-backup/2025-02-25T00:00:21Z logical ready 156d 156d
cron-prod-20250226000000-zrgwj prod minio s3://pmdb-prod-backup/2025-02-26T00:00:21Z logical ready 155d 155d
cron-prod-20250227000000-pn2dp prod minio s3://pmdb-prod-backup/2025-02-27T00:00:21Z logical ready 154d 154d
cron-prod-20250228000000-w49k5 prod minio s3://pmdb-prod-backup/2025-02-28T00:00:21Z logical ready 153d 153d
cron-prod-20250301000000-xrmqk prod minio s3://pmdb-prod-backup/2025-03-01T00:00:21Z logical ready 152d 152d
cron-prod-20250302000000-npflb prod minio s3://pmdb-prod-backup/2025-03-02T00:00:21Z logical ready 151d 151d
cron-prod-20250303000000-gv7mk prod minio s3://pmdb-prod-backup/2025-03-03T00:00:21Z logical ready 150d 150d
cron-prod-20250304000000-bsx8x prod minio s3://pmdb-prod-backup/2025-03-04T00:00:21Z logical ready 149d 149d
cron-prod-20250305000000-q2lsz prod minio s3://pmdb-prod-backup/2025-03-05T00:00:21Z logical ready 148d 148d
cron-prod-20250306000000-r9xgq prod minio s3://pmdb-prod-backup/2025-03-06T00:00:21Z logical ready 147d 147d
cron-prod-20250307000000-msx72 prod minio s3://pmdb-prod-backup/2025-03-07T00:00:21Z logical ready 146d 146d
cron-prod-20250308000000-zgmfj prod minio s3://pmdb-prod-backup/2025-03-08T00:00:21Z logical ready 145d 145d
cron-prod-20250309000000-4t4lz prod minio s3://pmdb-prod-backup/2025-03-09T00:00:21Z logical ready 144d 144d
cron-prod-20250310000000-qgblc prod minio s3://pmdb-prod-backup/2025-03-10T00:00:21Z logical ready 143d 143d
cron-prod-20250311000000-5wjhf prod minio s3://pmdb-prod-backup/2025-03-11T00:00:21Z logical ready 142d 142d
cron-prod-20250312000000-4slw7 prod minio s3://pmdb-prod-backup/2025-03-12T00:00:21Z logical ready 141d 141d
cron-prod-20250313000000-9fjjf prod minio s3://pmdb-prod-backup/2025-03-13T00:00:21Z logical ready 140d 140d
cron-prod-20250314000000-pcw79 prod minio s3://pmdb-prod-backup/2025-03-14T00:00:21Z logical ready 139d 139d
cron-prod-20250315000000-bjlsq prod minio s3://pmdb-prod-backup/2025-03-15T00:00:21Z logical ready 138d 138d
cron-prod-20250316000000-pkqhm prod minio s3://pmdb-prod-backup/2025-03-16T00:00:21Z logical ready 137d 137d
cron-prod-20250317000000-l7g5m prod minio s3://pmdb-prod-backup/2025-03-17T00:00:21Z logical ready 136d 136d
cron-prod-20250318000000-sgqrl prod minio s3://pmdb-prod-backup/2025-03-18T00:00:21Z logical ready 135d 135d
cron-prod-20250319000000-5hwkm prod minio s3://pmdb-prod-backup/2025-03-19T00:00:21Z logical ready 134d 134d
cron-prod-20250320000000-nmtcv prod minio s3://pmdb-prod-backup/2025-03-20T00:00:21Z logical ready 133d 133d
cron-prod-20250321000000-rgx85 prod minio s3://pmdb-prod-backup/2025-03-21T00:00:21Z logical ready 132d 132d
cron-prod-20250322000000-fr966 prod minio s3://pmdb-prod-backup/2025-03-22T00:00:21Z logical ready 131d 131d
cron-prod-20250323000000-htlmd prod minio s3://pmdb-prod-backup/2025-03-23T00:00:21Z logical ready 130d 130d
cron-prod-20250324000000-sd2hx prod minio s3://pmdb-prod-backup/2025-03-24T00:00:21Z logical ready 129d 129d
cron-prod-20250325000000-tnf8b prod minio s3://pmdb-prod-backup/2025-03-25T00:00:21Z logical ready 128d 128d
...
cron-prod-20250510000000-7pqc4 prod minio s3://pmdb-prod-backup/2025-05-10T00:00:21Z logical ready 82d 82d
cron-prod-20250511000000-qkzcc prod minio s3://pmdb-prod-backup/2025-05-11T00:00:21Z logical ready 81d 81d
cron-prod-20250512000000-gccsn prod minio s3://pmdb-prod-backup/2025-05-12T00:00:21Z logical ready 80d 80d
cron-prod-20250513000000-sscsn prod minio s3://pmdb-prod-backup/2025-05-13T00:00:21Z logical ready 79d 79d
cron-prod-20250514000000-4m6qn prod minio s3://pmdb-prod-backup/2025-05-14T00:00:21Z logical ready 78d 78d
cron-prod-20250515000000-r6lj5 prod minio s3://pmdb-prod-backup/2025-05-15T00:00:21Z logical ready 77d 77d
cron-prod-20250516000000-24m7n prod minio s3://pmdb-prod-backup/2025-05-16T00:00:21Z logical ready 76d 76d
cron-prod-20250517000000-bk2lj prod minio s3://pmdb-prod-backup/2025-05-17T00:00:21Z logical ready 75d 75d
cron-prod-20250518000000-99g9s prod minio s3://pmdb-prod-backup/2025-05-18T00:00:21Z logical ready 74d 74d
cron-prod-20250519000000-wsccr prod minio s3://pmdb-prod-backup/2025-05-19T00:00:21Z logical ready 73d 73d
cron-prod-20250520000000-c4fkc prod minio s3://pmdb-prod-backup/2025-05-20T00:00:21Z logical ready 72d 72d
cron-prod-20250521000000-slmvl prod minio s3://pmdb-prod-backup/2025-05-21T00:00:21Z logical ready 71d 71d
cron-prod-20250522000000-hr5tx prod minio s3://pmdb-prod-backup/2025-05-22T00:00:21Z logical ready 70d 70d
cron-prod-20250523000000-k462p prod minio s3://pmdb-prod-backup/2025-05-23T00:00:21Z logical ready 69d 69d
cron-prod-20250524000000-tv6cj prod minio s3://pmdb-prod-backup/2025-05-24T00:00:21Z logical ready 68d 68d
cron-prod-20250525000000-8vgt8 prod minio s3://pmdb-prod-backup/2025-05-25T00:00:21Z logical ready 67d 67d
cron-prod-20250526000000-4hgcn prod minio s3://pmdb-prod-backup/2025-05-26T00:00:21Z logical ready 66d 66d
cron-prod-20250527000000-56j4k prod minio s3://pmdb-prod-backup/2025-05-27T00:00:21Z logical ready 65d 65d
cron-prod-20250528000000-vhpww prod minio s3://pmdb-prod-backup/2025-05-28T00:00:21Z logical ready 64d 64d
cron-prod-20250529000000-2w2k5 prod minio s3://pmdb-prod-backup/2025-05-29T00:00:21Z logical ready 63d 63d
cron-prod-20250530000000-s6d7c prod minio s3://pmdb-prod-backup/2025-05-30T00:00:21Z logical ready 62d 62d
cron-prod-20250531000000-28gbz prod minio s3://pmdb-prod-backup/2025-05-31T00:00:21Z logical ready 61d 61d
cron-prod-20250601000000-txtxs prod minio s3://pmdb-prod-backup/2025-06-01T00:00:21Z logical ready 60d 60d
cron-prod-20250602000000-86dtz prod minio s3://pmdb-prod-backup/2025-06-02T00:00:21Z logical ready 59d 59d
cron-prod-20250603000000-f27gx prod minio s3://pmdb-prod-backup/2025-06-03T00:00:21Z logical ready 58d 58d
cron-prod-20250604000000-hhs6n prod minio s3://pmdb-prod-backup/2025-06-04T00:00:21Z logical ready 57d 57d
cron-prod-20250605000000-ll45p prod minio s3://pmdb-prod-backup/2025-06-05T00:00:21Z logical ready 56d 56d
cron-prod-20250606000000-f57lz prod minio s3://pmdb-prod-backup/2025-06-06T00:00:21Z logical ready 55d 55d
cron-prod-20250607000000-f9fss prod minio s3://pmdb-prod-backup/2025-06-07T00:00:21Z logical ready 54d 54d
cron-prod-20250608000000-z22zs prod minio s3://pmdb-prod-backup/2025-06-08T00:00:21Z logical ready 53d 53d
cron-prod-20250609000000-xpgmg prod minio s3://pmdb-prod-backup/2025-06-09T00:00:21Z logical ready 52d 52d
cron-prod-20250610000000-k95q6 prod minio s3://pmdb-prod-backup/2025-06-10T00:00:21Z logical ready 51d 51d
cron-prod-20250611000000-k6bgm prod minio s3://pmdb-prod-backup/2025-06-11T00:00:21Z logical ready 50d 50d
cron-prod-20250612000000-s5gv4 prod minio s3://pmdb-prod-backup/2025-06-12T00:00:21Z logical ready 49d 49d
cron-prod-20250613000000-2jzq6 prod minio s3://pmdb-prod-backup/2025-06-13T00:00:21Z logical ready 48d 48d
cron-prod-20250614000000-jkmrd prod minio s3://pmdb-prod-backup/2025-06-14T00:00:21Z logical ready 47d 47d
cron-prod-20250615000000-h9kg8 prod minio s3://pmdb-prod-backup/2025-06-15T00:00:21Z logical ready 46d 46d
cron-prod-20250616000000-fqp48 prod minio s3://pmdb-prod-backup/2025-06-16T00:00:21Z logical ready 45d 45d
cron-prod-20250617000000-64tmx prod minio s3://pmdb-prod-backup/2025-06-17T00:00:21Z logical ready 44d 44d
cron-prod-20250618000000-mmk5v prod minio s3://pmdb-prod-backup/2025-06-18T00:00:21Z logical ready 43d 43d
cron-prod-20250619000000-jckw7 prod minio s3://pmdb-prod-backup/2025-06-19T00:00:21Z logical ready 42d 42d
cron-prod-20250620000000-qp7ps prod minio s3://pmdb-prod-backup/2025-06-20T00:00:21Z logical ready 41d 41d
cron-prod-20250621000000-2qpxd prod minio s3://pmdb-prod-backup/2025-06-21T00:00:21Z logical ready 40d 40d
cron-prod-20250622000000-n9fx5 prod minio s3://pmdb-prod-backup/2025-06-22T00:00:21Z logical ready 39d 39d
cron-prod-20250623000000-kbj7t prod minio s3://pmdb-prod-backup/2025-06-23T00:00:21Z logical ready 38d 38d
cron-prod-20250624000000-9db66 prod minio s3://pmdb-prod-backup/2025-06-24T00:00:21Z logical ready 37d 37d
cron-prod-20250625000000-2qwkg prod minio s3://pmdb-prod-backup/2025-06-25T00:00:21Z logical ready 36d 36d
cron-prod-20250626000000-fz22n prod minio s3://pmdb-prod-backup/2025-06-26T00:00:21Z logical ready 35d 35d
cron-prod-20250627000000-rbffp prod minio s3://pmdb-prod-backup/2025-06-27T00:00:21Z logical ready 34d 34d
cron-prod-20250628000000-slldt prod minio s3://pmdb-prod-backup/2025-06-28T00:00:21Z logical ready 33d 33d
cron-prod-20250629000000-5qqdc prod minio s3://pmdb-prod-backup/2025-06-29T00:00:21Z logical ready 32d 32d
cron-prod-20250630000000-g2c5b prod minio s3://pmdb-prod-backup/2025-06-30T00:00:21Z logical ready 31d 31d
cron-prod-20250701000000-k82lh prod minio s3://pmdb-prod-backup/2025-07-01T00:00:21Z logical ready 30d 30d
cron-prod-20250702000000-65h4k prod minio s3://pmdb-prod-backup/2025-07-02T00:00:21Z logical ready 29d 29d
cron-prod-20250703000000-zm799 prod minio s3://pmdb-prod-backup/2025-07-03T00:00:21Z logical ready 28d 28d
cron-prod-20250704000000-jxkbm prod minio s3://pmdb-prod-backup/2025-07-04T00:00:21Z logical ready 27d 27d
cron-prod-20250705000000-bxh27 prod minio s3://pmdb-prod-backup/2025-07-05T00:00:21Z logical ready 26d 26d
cron-prod-20250706000000-dtrsk prod minio s3://pmdb-prod-backup/2025-07-06T00:00:21Z logical ready 25d 25d
cron-prod-20250707000000-rq72q prod minio s3://pmdb-prod-backup/2025-07-07T00:00:21Z logical ready 24d 24d
cron-prod-20250708000000-xf5pm prod minio s3://pmdb-prod-backup/2025-07-08T00:00:21Z logical ready 23d 23d
cron-prod-20250709000000-8lwgw prod minio s3://pmdb-prod-backup/2025-07-09T00:00:21Z logical ready 22d 22d
cron-prod-20250710000000-vrn7x prod minio s3://pmdb-prod-backup/2025-07-10T00:00:21Z logical ready 21d 21d
cron-prod-20250711000000-h9g5c prod minio s3://pmdb-prod-backup/2025-07-11T00:00:21Z logical ready 20d 20d
cron-prod-20250712000000-6fsnt prod minio s3://pmdb-prod-backup/2025-07-12T00:00:21Z logical ready 19d 19d
cron-prod-20250713000000-nftwc prod minio s3://pmdb-prod-backup/2025-07-13T00:00:21Z logical ready 18d 18d
cron-prod-20250714000000-s2x7x prod minio s3://pmdb-prod-backup/2025-07-14T00:00:21Z logical ready 17d 17d
cron-prod-20250715000000-xkbmc prod minio s3://pmdb-prod-backup/2025-07-15T00:00:21Z logical ready 16d 16d
cron-prod-20250716000000-9qx9w prod minio s3://pmdb-prod-backup/2025-07-16T00:00:21Z logical ready 15d 15d
cron-prod-20250717000000-7l264 prod minio s3://pmdb-prod-backup/2025-07-17T00:00:21Z logical ready 14d 14d
cron-prod-20250718000000-xb6pk prod minio s3://pmdb-prod-backup/2025-07-18T00:00:21Z logical ready 13d 13d
cron-prod-20250719000000-zqh4m prod minio s3://pmdb-prod-backup/2025-07-19T00:00:21Z logical ready 12d 12d
cron-prod-20250720000000-lwnp4 prod minio s3://pmdb-prod-backup/2025-07-20T00:00:21Z logical ready 11d 11d
cron-prod-20250721000000-kdrjh prod minio s3://pmdb-prod-backup/2025-07-21T00:00:21Z logical ready 10d 10d
cron-prod-20250722000000-xg8x9 prod minio s3://pmdb-prod-backup/2025-07-22T00:00:21Z logical ready 9d 9d
cron-prod-20250723000000-fhxn2 prod minio s3://pmdb-prod-backup/2025-07-23T00:00:21Z logical ready 8d 8d
cron-prod-20250724000000-9n6x6 prod minio s3://pmdb-prod-backup/2025-07-24T00:00:21Z logical ready 7d13h 7d13h
cron-prod-20250725000000-gpg5f prod minio s3://pmdb-prod-backup/2025-07-25T00:00:21Z logical ready 6d13h 6d13h
cron-prod-20250726000000-lxvg6 prod minio s3://pmdb-prod-backup/2025-07-26T00:00:21Z logical ready 5d13h 5d13h
cron-prod-20250727000000-7n9s9 prod minio s3://pmdb-prod-backup/2025-07-27T00:00:21Z logical ready 4d13h 4d13h
...

```

Current logs:

```auto
oc -n prod-pmdb logs percona-server-mongodb-operator-65cd6cfc48-ktwds
2025-07-30T22:07:47.605Z INFO setup Manager starting up {"gitCommit": "5019408f1fe40483fc5effaf61ab3f672765b189", "gitBranch": "release-1-17-0", "goVersion": "go1.22.6", "os": "linux", "arch": "amd64"}
2025-07-30T22:07:47.616Z INFO server version {"platform": "openshift", "version": "undefined (v4.0+)"}
2025-07-30T22:07:47.631Z INFO controller-runtime.metrics Starting metrics server
2025-07-30T22:07:47.631Z INFO starting server {"name": "health probe", "addr": "[::]:8081"}
2025-07-30T22:07:47.631Z INFO controller-runtime.metrics Serving metrics server {"bindAddress": ":8080", "secure": false}
I0730 22:07:47.631676 1 leaderelection.go:250] attempting to acquire leader lease prod-pmdb/08db0feb.percona.com...
I0730 22:08:03.851549 1 leaderelection.go:260] successfully acquired lease prod-pmdb/08db0feb.percona.com
2025-07-30T22:08:03.851Z INFO Starting EventSource {"controller": "psmdbrestore-controller", "source": "kind source: *v1.PerconaServerMongoDBRestore"}
2025-07-30T22:08:03.851Z INFO Starting EventSource {"controller": "psmdbrestore-controller", "source": "kind source: *v1.Pod"}
2025-07-30T22:08:03.851Z INFO Starting Controller {"controller": "psmdbrestore-controller"}
2025-07-30T22:08:03.851Z INFO Starting EventSource {"controller": "psmdbbackup-controller", "source": "kind source: *v1.PerconaServerMongoDBBackup"}
2025-07-30T22:08:03.851Z INFO Starting EventSource {"controller": "psmdbbackup-controller", "source": "kind source: *v1.Pod"}
2025-07-30T22:08:03.851Z INFO Starting Controller {"controller": "psmdbbackup-controller"}
2025-07-30T22:08:03.851Z INFO Starting EventSource {"controller": "psmdb-controller", "source": "kind source: *v1.PerconaServerMongoDB"}
2025-07-30T22:08:03.851Z INFO Starting Controller {"controller": "psmdb-controller"}
2025-07-30T22:08:03.971Z INFO Starting workers {"controller": "psmdbrestore-controller", "worker count": 1}
2025-07-30T22:08:03.972Z INFO Starting workers {"controller": "psmdb-controller", "worker count": 1}
2025-07-30T22:08:03.972Z INFO Starting workers {"controller": "psmdbbackup-controller", "worker count": 1}
2025-07-30T22:08:03.999Z INFO Creating or updating backup job {"controller": "psmdb-controller", "object": {"name":"prod","namespace":"prod-pmdb"}, "namespace": "prod-pmdb", "name": "prod", "reconcileID": "83d3f4ff-b856-4965-b8f3-4cdd0b00f5c9", "name": "daily-backup", "namespace": "prod-pmdb", "schedule": "0 0 * * *"}
2025-07-30T22:08:04.191Z INFO add new job {"controller": "psmdb-controller", "object": {"name":"prod","namespace":"prod-pmdb"}, "namespace": "prod-pmdb", "name": "prod", "reconcileID": "83d3f4ff-b856-4965-b8f3-4cdd0b00f5c9", "name": "ensure-version/prod-pmdb/prod", "schedule": "0 2 * * *"}
2025-07-31T00:00:10.285Z INFO Starting backup {"controller": "psmdbbackup-controller", "object": {"name":"cron-prod-20250731000000-r6hs6","namespace":"prod-pmdb"}, "namespace": "prod-pmdb", "name": "cron-prod-20250731000000-r6hs6", "reconcileID": "48ad84f1-4b9d-4278-8fe0-88ffbc0453fc", "backup": "cron-prod-20250731000000-r6hs6", "storage": "minio"}
2025-07-31T00:00:10.285Z INFO Setting PBM config {"controller": "psmdbbackup-controller", "object": {"name":"cron-prod-20250731000000-r6hs6","namespace":"prod-pmdb"}, "namespace": "prod-pmdb", "name": "cron-prod-20250731000000-r6hs6", "reconcileID": "48ad84f1-4b9d-4278-8fe0-88ffbc0453fc", "backup": "prod"}
2025-07-31T00:00:21.292Z INFO Sending backup command {"controller": "psmdbbackup-controller", "object": {"name":"cron-prod-20250731000000-r6hs6","namespace":"prod-pmdb"}, "namespace": "prod-pmdb", "name": "cron-prod-20250731000000-r6hs6", "reconcileID": "48ad84f1-4b9d-4278-8fe0-88ffbc0453fc", "backupCmd": "backup [name: 2025-07-31T00:00:21Z, compression: gzip (level: default)] <ts: 0>"}
2025-07-31T13:15:50.875Z INFO Cluster state changed {"controller": "psmdb-controller", "object": {"name":"prod","namespace":"prod-pmdb"}, "namespace": "prod-pmdb", "name": "prod", "reconcileID": "3ff8ee71-81ee-4134-88b8-bb89c9e88a7b", "previous": "ready", "current": "initializing"}
2025-07-31T13:16:11.874Z INFO Cluster state changed {"controller": "psmdb-controller", "object": {"name":"prod","namespace":"prod-pmdb"}, "namespace": "prod-pmdb", "name": "prod", "reconcileID": "8517fb39-cd33-430b-924d-74e92e43e7e7", "previous": "initializing", "current": "ready"}
2025-07-31T13:24:39.943Z INFO Setting PBM config {"controller": "psmdbbackup-controller", "object": {"name":"cron-prod-20241116000000-wcj8f","namespace":"prod-pmdb"}, "namespace": "prod-pmdb", "name": "cron-prod-20241116000000-wcj8f", "reconcileID": "8f558153-0421-45a0-bb96-fb28032e082d", "backup": "prod"}
2025-07-31T13:24:51.004Z INFO nothing to delete {"controller": "psmdbbackup-controller", "object": {"name":"cron-prod-20241116000000-wcj8f","namespace":"prod-pmdb"}, "namespace": "prod-pmdb", "name": "cron-prod-20241116000000-wcj8f", "reconcileID": "8f558153-0421-45a0-bb96-fb28032e082d"}

```

---

<div class="post-metadata">

**Author:** ![SuperHammer](https://avatars.discourse-cdn.com/v4/letter/s/ccd318/32.png) [@SuperHammer](https://forums.percona.com/u/SuperHammer)\
**Post date:** [July 31, 2025, 1:42pm UTC](https://forums.percona.com/t/backup-retention-not-applied-in-v1-17-0/38904/3 "2025-07-31T13:42:11Z")

</div>

Logs before restart:

```auto
oc -n prod-pmdb logs --previous percona-server-mongodb-operator-65cd6cfc48-ktwds
goroutine 191 [select, 1867 minutes]:
reflect.rselect({0xc000e0df00, 0x3, 0x0?})
        /usr/local/go/src/runtime/select.go:589 +0x2c5
reflect.Select({0xc0008c2420?, 0x3, 0xc000cab7d0?})
        /usr/local/go/src/reflect/value.go:3169 +0x5ca
sigs.k8s.io/controller-runtime/pkg/internal/syncs.MergeChans[...].func2()
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/syncs/syncs.go:34 +0x55
created by sigs.k8s.io/controller-runtime/pkg/internal/syncs.MergeChans[...] in goroutine 181
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/syncs/syncs.go:32 +0x438

goroutine 193 [chan receive, 1867 minutes]:
k8s.io/client-go/tools/cache.(*sharedProcessor).run(0xc00088e640, 0xc000bc91a0)
        /go/pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/shared_informer.go:807 +0x4d
k8s.io/client-go/tools/cache.(*sharedIndexInformer).Run.(*Group).StartWithChannel.func4()
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/util/wait/wait.go:55 +0x1b
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/util/wait/wait.go:72 +0x52
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start in goroutine 181
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/util/wait/wait.go:70 +0x73

goroutine 194 [chan receive, 1867 minutes]:
k8s.io/client-go/tools/cache.(*controller).Run.func1()
        /go/pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/controller.go:132 +0x25
created by k8s.io/client-go/tools/cache.(*controller).Run in goroutine 181
        /go/pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/controller.go:131 +0xa9

goroutine 195 [select]:
k8s.io/client-go/tools/cache.watchHandler({0x0?, 0x0?, 0x3b5cf60?}, {0x29c0bd0, 0xc0003eda80}, {0x7f955fd08b28, 0xc0008c24d0}, {0x29f1138, 0x21ea180}, 0x0, ...)
        /go/pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/reflector.go:732 +0x187
k8s.io/client-go/tools/cache.(*Reflector).watch(0xc0008c4380, {0x0?, 0x0?}, 0xc000bc90e0, 0xc000bd4a20)
        /go/pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/reflector.go:445 +0x545
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch(0xc0008c4380, 0xc000bc90e0)
        /go/pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/reflector.go:365 +0x4d7
k8s.io/client-go/tools/cache.(*Reflector).Run.func1()
        /go/pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/reflector.go:298 +0x25
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x10?)
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/util/wait/backoff.go:226 +0x33
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc0005ddf50, {0x29bb420, 0xc00088e820}, 0x1, 0xc000bc90e0)
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/util/wait/backoff.go:227 +0xaf
k8s.io/client-go/tools/cache.(*Reflector).Run(0xc0008c4380, 0xc000bc90e0)
        /go/pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/reflector.go:297 +0x1c5
k8s.io/client-go/tools/cache.(*controller).Run.(*Group).StartWithChannel.func2()
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/util/wait/wait.go:55 +0x1b
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/util/wait/wait.go:72 +0x52
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start in goroutine 181
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/util/wait/wait.go:70 +0x73

goroutine 73 [select, 16 minutes]:
k8s.io/client-go/tools/cache.(*Reflector).startResync(0xc0008c4380, 0xc000bc90e0, 0xc0000b56e0, 0xc000bd4a20)
        /go/pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/reflector.go:377 +0x10f
created by k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch in goroutine 195
        /go/pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/reflector.go:364 +0x4ad

goroutine 172 [select, 1867 minutes]:
reflect.rselect({0xc000eaaf00, 0x3, 0x0?})
        /usr/local/go/src/runtime/select.go:589 +0x2c5
reflect.Select({0xc000c27b80?, 0x3, 0xc000ca7fd0?})
        /usr/local/go/src/reflect/value.go:3169 +0x5ca
sigs.k8s.io/controller-runtime/pkg/internal/syncs.MergeChans[...].func2()
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/syncs/syncs.go:34 +0x55
created by sigs.k8s.io/controller-runtime/pkg/internal/syncs.MergeChans[...] in goroutine 169
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/syncs/syncs.go:32 +0x438

goroutine 174 [chan receive, 1867 minutes]:
k8s.io/client-go/tools/cache.(*sharedProcessor).run(0xc000bfd360, 0xc00073b020)
        /go/pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/shared_informer.go:807 +0x4d
k8s.io/client-go/tools/cache.(*sharedIndexInformer).Run.(*Group).StartWithChannel.func4()
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/util/wait/wait.go:55 +0x1b
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/util/wait/wait.go:72 +0x52
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start in goroutine 169
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/util/wait/wait.go:70 +0x73

goroutine 175 [chan receive, 1867 minutes]:
k8s.io/client-go/tools/cache.(*controller).Run.func1()
        /go/pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/controller.go:132 +0x25
created by k8s.io/client-go/tools/cache.(*controller).Run in goroutine 169
        /go/pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/controller.go:131 +0xa9

goroutine 176 [select]:
k8s.io/client-go/tools/cache.watchHandler({0x0?, 0x0?, 0x3b5cf60?}, {0x29c0bd0, 0xc000905040}, {0x7f955fd08b28, 0xc000c27c30}, {0x29f1138, 0x21f6980}, 0x0, ...)
        /go/pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/reflector.go:732 +0x187
k8s.io/client-go/tools/cache.(*Reflector).watch(0xc0007da380, {0x0?, 0x0?}, 0xc00073af60, 0xc000563680)
        /go/pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/reflector.go:445 +0x545
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch(0xc0007da380, 0xc00073af60)
        /go/pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/reflector.go:365 +0x4d7
k8s.io/client-go/tools/cache.(*Reflector).Run.func1()
        /go/pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/reflector.go:298 +0x25
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x10?)
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/util/wait/backoff.go:226 +0x33
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc0005e1f50, {0x29bb420, 0xc000bfd4a0}, 0x1, 0xc00073af60)
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/util/wait/backoff.go:227 +0xaf
k8s.io/client-go/tools/cache.(*Reflector).Run(0xc0007da380, 0xc00073af60)
        /go/pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/reflector.go:297 +0x1c5
k8s.io/client-go/tools/cache.(*controller).Run.(*Group).StartWithChannel.func2()
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/util/wait/wait.go:55 +0x1b
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/util/wait/wait.go:72 +0x52
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start in goroutine 169
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/util/wait/wait.go:70 +0x73

goroutine 211 [select, 153 minutes]:
k8s.io/client-go/tools/cache.(*Reflector).startResync(0xc0007da380, 0xc00073af60, 0xc00073b3e0, 0xc000563680)
        /go/pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/reflector.go:377 +0x10f
created by k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch in goroutine 176
        /go/pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/reflector.go:364 +0x4ad

goroutine 231 [chan receive, 153 minutes]:
k8s.io/client-go/tools/cache.(*processorListener).run.func1()
        /go/pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/shared_informer.go:973 +0x4b
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/util/wait/backoff.go:226 +0x33
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000127f70, {0x29bb400, 0xc00120c0f0}, 0x1, 0xc0000b4f00)
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/util/wait/backoff.go:227 +0xaf
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000ca7f70, 0x3b9aca00, 0x0, 0x1, 0xc0000b4f00)
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/util/wait/backoff.go:204 +0x7f
k8s.io/apimachinery/pkg/util/wait.Until(...)
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/util/wait/backoff.go:161
k8s.io/client-go/tools/cache.(*processorListener).run(0xc000962000)
        /go/pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/shared_informer.go:972 +0x69
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/util/wait/wait.go:72 +0x52
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start in goroutine 136
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/util/wait/wait.go:70 +0x73

goroutine 232 [select, 153 minutes]:
k8s.io/client-go/tools/cache.(*processorListener).pop(0xc000962000)
        /go/pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/shared_informer.go:943 +0x107
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/util/wait/wait.go:72 +0x52
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start in goroutine 136
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/util/wait/wait.go:70 +0x73

<similar logs omitted due to post size>

goroutine 2016071 [sync.Cond.Wait, 1327 minutes]:
sync.runtime_notifyListWait(0xc000c26238, 0x1)
        /usr/local/go/src/runtime/sema.go:569 +0x159
sync.(*Cond).Wait(0xc000537a00?)
        /usr/local/go/src/sync/cond.go:70 +0x85
k8s.io/client-go/tools/cache.(*DeltaFIFO).Pop(0xc000c26210, 0xc000223eb0)
        /go/pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/delta_fifo.go:575 +0x236
k8s.io/client-go/tools/cache.(*controller).processLoop(0xc000636460)
        /go/pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/controller.go:188 +0x30
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/util/wait/backoff.go:226 +0x33
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000e42df0, {0x29bb400, 0xc000e48870}, 0x1, 0xc000c6a600)
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/util/wait/backoff.go:227 +0xaf
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000e42df0, 0x3b9aca00, 0x0, 0x1, 0xc000c6a600)
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/util/wait/backoff.go:204 +0x7f
k8s.io/apimachinery/pkg/util/wait.Until(...)
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/util/wait/backoff.go:161
k8s.io/client-go/tools/cache.(*controller).Run(0xc000636460, 0xc000c6a600)
        /go/pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/controller.go:159 +0x35e
k8s.io/client-go/tools/cache.(*sharedIndexInformer).Run(0xc001068790, 0xc000c6a600)
        /go/pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/shared_informer.go:508 +0x2c8
sigs.k8s.io/controller-runtime/pkg/cache/internal.(*Cache).Start(0xc000290690, 0xc00163d7d0?)
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/cache/internal/informers.go:106 +0x72
sigs.k8s.io/controller-runtime/pkg/cache/internal.(*Informers).startInformerLocked.func1()
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/cache/internal/informers.go:239 +0x7b
created by sigs.k8s.io/controller-runtime/pkg/cache/internal.(*Informers).startInformerLocked in goroutine 144
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/cache/internal/informers.go:237 +0x87

goroutine 2015993 [sync.Cond.Wait, 1327 minutes]:
sync.runtime_notifyListWait(0xc000b7e398, 0x1)
        /usr/local/go/src/runtime/sema.go:569 +0x159
sync.(*Cond).Wait(0xc0005372a0?)
        /usr/local/go/src/sync/cond.go:70 +0x85
k8s.io/client-go/tools/cache.(*DeltaFIFO).Pop(0xc000b7e370, 0xc00071e5d0)
        /go/pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/delta_fifo.go:575 +0x236
k8s.io/client-go/tools/cache.(*controller).processLoop(0xc000724280)
        /go/pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/controller.go:188 +0x30
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/util/wait/backoff.go:226 +0x33
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000128df0, {0x29bb400, 0xc000a862a0}, 0x1, 0xc000bc82a0)
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/util/wait/backoff.go:227 +0xaf
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000128df0, 0x3b9aca00, 0x0, 0x1, 0xc000bc82a0)
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/util/wait/backoff.go:204 +0x7f
k8s.io/apimachinery/pkg/util/wait.Until(...)
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/util/wait/backoff.go:161
k8s.io/client-go/tools/cache.(*controller).Run(0xc000724280, 0xc000bc82a0)
        /go/pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/controller.go:159 +0x35e
k8s.io/client-go/tools/cache.(*sharedIndexInformer).Run(0xc000b7e000, 0xc000bc82a0)
        /go/pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/shared_informer.go:508 +0x2c8
sigs.k8s.io/controller-runtime/pkg/cache/internal.(*Cache).Start(0xc0006147e0, 0xc000ca9fd0?)
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/cache/internal/informers.go:106 +0x72
sigs.k8s.io/controller-runtime/pkg/cache/internal.(*Informers).startInformerLocked.func1()
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/cache/internal/informers.go:239 +0x7b
created by sigs.k8s.io/controller-runtime/pkg/cache/internal.(*Informers).startInformerLocked in goroutine 144
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/cache/internal/informers.go:237 +0x87

goroutine 2015998 [chan receive, 1327 minutes]:
k8s.io/client-go/tools/cache.(*controller).Run.func1()
        /go/pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/controller.go:132 +0x25
created by k8s.io/client-go/tools/cache.(*controller).Run in goroutine 2015993
        /go/pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/controller.go:131 +0xa9

goroutine 2015995 [select, 1327 minutes]:
reflect.rselect({0xc001031f00, 0x3, 0xc001031e90?})
        /usr/local/go/src/runtime/select.go:589 +0x2c5
reflect.Select({0xc000b7e2c0?, 0x3, 0xc001031fd0?})
        /usr/local/go/src/reflect/value.go:3169 +0x5ca
sigs.k8s.io/controller-runtime/pkg/internal/syncs.MergeChans[...].func2()
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/syncs/syncs.go:34 +0x55
created by sigs.k8s.io/controller-runtime/pkg/internal/syncs.MergeChans[...] in goroutine 2015993
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/syncs/syncs.go:32 +0x438

goroutine 2016098 [select, 65 minutes]:
k8s.io/client-go/tools/cache.(*Reflector).startResync(0xc0015b68c0, 0xc000bc82a0, 0xc000bc8780, 0xc0014c7080)
        /go/pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/reflector.go:377 +0x10f
created by k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch in goroutine 2015999
        /go/pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/reflector.go:364 +0x4ad

goroutine 6955599 [sync.Cond.Wait]:
sync.runtime_notifyListWait(0xc000b4ac48, 0x5)
        /usr/local/go/src/runtime/sema.go:569 +0x159
sync.(*Cond).Wait(0xc000bb0dc0?)
        /usr/local/go/src/sync/cond.go:70 +0x85
golang.org/x/net/http2.(*pipe).Read(0xc000b4ac30, {0xc0011a0601, 0x5ff, 0x5ff})
        /go/pkg/mod/golang.org/x/net@v0.26.0/http2/pipe.go:76 +0xdf
golang.org/x/net/http2.transportResponseBody.Read({0x0?}, {0xc0011a0601?, 0xc000851ce0?, 0x411bbb?})
        /go/pkg/mod/golang.org/x/net@v0.26.0/http2/transport.go:2641 +0x65
encoding/json.(*Decoder).refill(0xc001302500)
        /usr/local/go/src/encoding/json/stream.go:165 +0x188
encoding/json.(*Decoder).readValue(0xc001302500)
        /usr/local/go/src/encoding/json/stream.go:140 +0x85
encoding/json.(*Decoder).Decode(0xc001302500, {0x1f414a0, 0xc000a9efd8})
        /usr/local/go/src/encoding/json/stream.go:63 +0x75
k8s.io/apimachinery/pkg/util/framer.(*jsonFrameReader).Read(0xc000c0d770, {0xc000bf8400, 0x400, 0x400})
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/util/framer/framer.go:152 +0x19c
k8s.io/apimachinery/pkg/runtime/serializer/streaming.(*decoder).Decode(0xc000bfc730, 0x0, {0x29bfff0, 0xc001165b00})
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/runtime/serializer/streaming/streaming.go:77 +0xa3
k8s.io/client-go/rest/watch.(*Decoder).Decode(0xc000fa63e0)
        /go/pkg/mod/k8s.io/client-go@v0.30.3/rest/watch/decoder.go:49 +0x4b
k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive(0xc000bb0e40)
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/watch/streamwatcher.go:105 +0xdb
created by k8s.io/apimachinery/pkg/watch.NewStreamWatcher in goroutine 188
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/watch/streamwatcher.go:76 +0x105

goroutine 6955598 [select, 5 minutes]:
golang.org/x/net/http2.(*clientStream).writeRequest(0xc000b4ac00, 0xc000bfa120, 0x0)
        /go/pkg/mod/golang.org/x/net@v0.26.0/http2/transport.go:1536 +0xa85
golang.org/x/net/http2.(*clientStream).doRequest(0xc000b4ac00, 0x921ce5?, 0xc000bb0c00?)
        /go/pkg/mod/golang.org/x/net@v0.26.0/http2/transport.go:1414 +0x56
created by golang.org/x/net/http2.(*ClientConn).roundTrip in goroutine 188
        /go/pkg/mod/golang.org/x/net@v0.26.0/http2/transport.go:1319 +0x3e5

goroutine 6952073 [sync.Cond.Wait]:
sync.runtime_notifyListWait(0xc000a6e348, 0x6)
        /usr/local/go/src/runtime/sema.go:569 +0x159
sync.(*Cond).Wait(0xc000bb1a40?)
        /usr/local/go/src/sync/cond.go:70 +0x85
golang.org/x/net/http2.(*pipe).Read(0xc000a6e330, {0xc000f94001, 0x5ff, 0x5ff})
        /go/pkg/mod/golang.org/x/net@v0.26.0/http2/pipe.go:76 +0xdf
golang.org/x/net/http2.transportResponseBody.Read({0x0?}, {0xc000f94001?, 0xc000853ce0?, 0x411bbb?})
        /go/pkg/mod/golang.org/x/net@v0.26.0/http2/transport.go:2641 +0x65
encoding/json.(*Decoder).refill(0xc000727b80)
        /usr/local/go/src/encoding/json/stream.go:165 +0x188
encoding/json.(*Decoder).readValue(0xc000727b80)
        /usr/local/go/src/encoding/json/stream.go:140 +0x85
encoding/json.(*Decoder).Decode(0xc000727b80, {0x1f414a0, 0xc000ba8cf0})
        /usr/local/go/src/encoding/json/stream.go:63 +0x75
k8s.io/apimachinery/pkg/util/framer.(*jsonFrameReader).Read(0xc000120960, {0xc0018f6000, 0x400, 0x400})
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/util/framer/framer.go:152 +0x19c
k8s.io/apimachinery/pkg/runtime/serializer/streaming.(*decoder).Decode(0xc000722960, 0x0, {0x29bfff0, 0xc00167ca40})
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/runtime/serializer/streaming/streaming.go:77 +0xa3
k8s.io/client-go/rest/watch.(*Decoder).Decode(0xc000fa77a0)
        /go/pkg/mod/k8s.io/client-go@v0.30.3/rest/watch/decoder.go:49 +0x4b
k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive(0xc000bb1ac0)
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/watch/streamwatcher.go:105 +0xdb
created by k8s.io/apimachinery/pkg/watch.NewStreamWatcher in goroutine 111
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/watch/streamwatcher.go:76 +0x105

goroutine 6971388 [sync.Cond.Wait]:
sync.runtime_notifyListWait(0xc001081848, 0x1)
        /usr/local/go/src/runtime/sema.go:569 +0x159
sync.(*Cond).Wait(0x0?)
        /usr/local/go/src/sync/cond.go:70 +0x85
golang.org/x/net/http2.(*pipe).Read(0xc001081830, {0xc0011a0c01, 0x5ff, 0x5ff})
        /go/pkg/mod/golang.org/x/net@v0.26.0/http2/pipe.go:76 +0xdf
golang.org/x/net/http2.transportResponseBody.Read({0x7f957fe5b798?}, {0xc0011a0c01?, 0xc00084fce0?, 0x411bbb?})
        /go/pkg/mod/golang.org/x/net@v0.26.0/http2/transport.go:2641 +0x65
encoding/json.(*Decoder).refill(0xc000e7fb80)
        /usr/local/go/src/encoding/json/stream.go:165 +0x188
encoding/json.(*Decoder).readValue(0xc000e7fb80)
        /usr/local/go/src/encoding/json/stream.go:140 +0x85
encoding/json.(*Decoder).Decode(0xc000e7fb80, {0x1f414a0, 0xc0002a5e00})
        /usr/local/go/src/encoding/json/stream.go:63 +0x75
k8s.io/apimachinery/pkg/util/framer.(*jsonFrameReader).Read(0xc000e49290, {0xc00109dc00, 0x400, 0x400})
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/util/framer/framer.go:152 +0x19c
k8s.io/apimachinery/pkg/runtime/serializer/streaming.(*decoder).Decode(0xc0007efc70, 0x0, {0x29bfff0, 0xc001458a40})
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/runtime/serializer/streaming/streaming.go:77 +0xa3
k8s.io/client-go/rest/watch.(*Decoder).Decode(0xc000885060)
        /go/pkg/mod/k8s.io/client-go@v0.30.3/rest/watch/decoder.go:49 +0x4b
k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive(0xc0003eda80)
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/watch/streamwatcher.go:105 +0xdb
created by k8s.io/apimachinery/pkg/watch.NewStreamWatcher in goroutine 195
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/watch/streamwatcher.go:76 +0x105

goroutine 6971387 [select, 1 minutes]:
golang.org/x/net/http2.(*clientStream).writeRequest(0xc001081800, 0xc001134ea0, 0x0)
        /go/pkg/mod/golang.org/x/net@v0.26.0/http2/transport.go:1536 +0xa85
golang.org/x/net/http2.(*clientStream).doRequest(0xc001081800, 0xc000784460?, 0xc00012d930?)
        /go/pkg/mod/golang.org/x/net@v0.26.0/http2/transport.go:1414 +0x56
created by golang.org/x/net/http2.(*ClientConn).roundTrip in goroutine 195
        /go/pkg/mod/golang.org/x/net@v0.26.0/http2/transport.go:1319 +0x3e5

goroutine 6972062 [select, 1 minutes]:
golang.org/x/net/http2.(*clientStream).writeRequest(0xc00133f800, 0xc00088a900, 0x0)
        /go/pkg/mod/golang.org/x/net@v0.26.0/http2/transport.go:1536 +0xa85
golang.org/x/net/http2.(*clientStream).doRequest(0xc00133f800, 0x921ce5?, 0xc00086bac0?)
        /go/pkg/mod/golang.org/x/net@v0.26.0/http2/transport.go:1414 +0x56
created by golang.org/x/net/http2.(*ClientConn).roundTrip in goroutine 2015999
        /go/pkg/mod/golang.org/x/net@v0.26.0/http2/transport.go:1319 +0x3e5

goroutine 2015997 [chan receive, 1327 minutes]:
k8s.io/client-go/tools/cache.(*sharedProcessor).run(0xc000168410, 0xc000bc83c0)
        /go/pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/shared_informer.go:807 +0x4d
k8s.io/client-go/tools/cache.(*sharedIndexInformer).Run.(*Group).StartWithChannel.func4()
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/util/wait/wait.go:55 +0x1b
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/util/wait/wait.go:72 +0x52
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start in goroutine 2015993
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/util/wait/wait.go:70 +0x73

goroutine 2015999 [select]:
k8s.io/client-go/tools/cache.watchHandler({0x0?, 0x0?, 0x3b5cf60?}, {0x29c0bd0, 0xc000de5c80}, {0x7f955fd08b28, 0xc000b7e370}, {0x29f1138, 0x21f1280}, 0x0, ...)
        /go/pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/reflector.go:732 +0x187
k8s.io/client-go/tools/cache.(*Reflector).watch(0xc0015b68c0, {0x0?, 0x0?}, 0xc000bc82a0, 0xc0014c7080)
        /go/pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/reflector.go:445 +0x545
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch(0xc0015b68c0, 0xc000bc82a0)
        /go/pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/reflector.go:365 +0x4d7
k8s.io/client-go/tools/cache.(*Reflector).Run.func1()
        /go/pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/reflector.go:298 +0x25
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x10?)
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/util/wait/backoff.go:226 +0x33
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc00084df50, {0x29bb420, 0xc000168640}, 0x1, 0xc000bc82a0)
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/util/wait/backoff.go:227 +0xaf
k8s.io/client-go/tools/cache.(*Reflector).Run(0xc0015b68c0, 0xc000bc82a0)
        /go/pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/reflector.go:297 +0x1c5
k8s.io/client-go/tools/cache.(*controller).Run.(*Group).StartWithChannel.func2()
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/util/wait/wait.go:55 +0x1b
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/util/wait/wait.go:72 +0x52
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start in goroutine 2015993
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/util/wait/wait.go:70 +0x73

goroutine 6955597 [sync.Cond.Wait]:
sync.runtime_notifyListWait(0xc000e8d548, 0x5)
        /usr/local/go/src/runtime/sema.go:569 +0x159
sync.(*Cond).Wait(0x2?)
        /usr/local/go/src/sync/cond.go:70 +0x85
golang.org/x/net/http2.(*pipe).Read(0xc000e8d530, {0xc001791498, 0x4, 0x4})
        /go/pkg/mod/golang.org/x/net@v0.26.0/http2/pipe.go:76 +0xdf
golang.org/x/net/http2.transportResponseBody.Read({0x0?}, {0xc001791498?, 0xc000fcbce0?, 0x411f65?})
        /go/pkg/mod/golang.org/x/net@v0.26.0/http2/transport.go:2641 +0x65
io.ReadAtLeast({0x7f95840c5818, 0xc000e8d500}, {0xc001791498, 0x4, 0x4}, 0x4)
        /usr/local/go/src/io/io.go:335 +0x90
k8s.io/apimachinery/pkg/util/framer.(*lengthDelimitedFrameReader).Read(0xc000ba8150, {0xc000bf8000, 0x400, 0x400})
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/util/framer/framer.go:76 +0x88
k8s.io/apimachinery/pkg/runtime/serializer/streaming.(*decoder).Decode(0xc000bfc690, 0x0, {0x29bfff0, 0xc0011650c0})
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/runtime/serializer/streaming/streaming.go:77 +0xa3
k8s.io/client-go/rest/watch.(*Decoder).Decode(0xc000fa61e0)
        /go/pkg/mod/k8s.io/client-go@v0.30.3/rest/watch/decoder.go:49 +0x4b
k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive(0xc000bb0c00)
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/watch/streamwatcher.go:105 +0xdb
created by k8s.io/apimachinery/pkg/watch.NewStreamWatcher in goroutine 2016113
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/watch/streamwatcher.go:76 +0x105

goroutine 6955596 [select, 5 minutes]:
golang.org/x/net/http2.(*clientStream).writeRequest(0xc000e8d500, 0xc0010f5b00, 0x0)
        /go/pkg/mod/golang.org/x/net@v0.26.0/http2/transport.go:1536 +0xa85
golang.org/x/net/http2.(*clientStream).doRequest(0xc000e8d500, 0xc000529540?, 0xc00071e260?)
        /go/pkg/mod/golang.org/x/net@v0.26.0/http2/transport.go:1414 +0x56
created by golang.org/x/net/http2.(*ClientConn).roundTrip in goroutine 2016113
        /go/pkg/mod/golang.org/x/net@v0.26.0/http2/transport.go:1319 +0x3e5

goroutine 2016113 [select]:
k8s.io/client-go/tools/cache.watchHandler({0x0?, 0x0?, 0x3b5cf60?}, {0x29c0bd0, 0xc000bb0c00}, {0x7f955fd08b28, 0xc000c26210}, {0x29f1138, 0x21f7f40}, 0x0, ...)
        /go/pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/reflector.go:732 +0x187
k8s.io/client-go/tools/cache.(*Reflector).watch(0xc00127b960, {0x0?, 0x0?}, 0xc000c6a600, 0xc0014c7920)
        /go/pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/reflector.go:445 +0x545
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch(0xc00127b960, 0xc000c6a600)
        /go/pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/reflector.go:365 +0x4d7
k8s.io/client-go/tools/cache.(*Reflector).Run.func1()
        /go/pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/reflector.go:298 +0x25
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x10?)
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/util/wait/backoff.go:226 +0x33
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000875f50, {0x29bb420, 0xc000118d70}, 0x1, 0xc000c6a600)
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/util/wait/backoff.go:227 +0xaf
k8s.io/client-go/tools/cache.(*Reflector).Run(0xc00127b960, 0xc000c6a600)
        /go/pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/reflector.go:297 +0x1c5
k8s.io/client-go/tools/cache.(*controller).Run.(*Group).StartWithChannel.func2()
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/util/wait/wait.go:55 +0x1b
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/util/wait/wait.go:72 +0x52
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start in goroutine 2016071
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/util/wait/wait.go:70 +0x73

goroutine 2016048 [chan receive, 1327 minutes]:
k8s.io/client-go/tools/cache.(*controller).Run.func1()
        /go/pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/controller.go:132 +0x25
created by k8s.io/client-go/tools/cache.(*controller).Run in goroutine 2016071
        /go/pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/controller.go:131 +0xa9

goroutine 2016047 [chan receive, 1327 minutes]:
k8s.io/client-go/tools/cache.(*sharedProcessor).run(0xc0000b2e10, 0xc000c6a6c0)
        /go/pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/shared_informer.go:807 +0x4d
k8s.io/client-go/tools/cache.(*sharedIndexInformer).Run.(*Group).StartWithChannel.func4()
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/util/wait/wait.go:55 +0x1b
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/util/wait/wait.go:72 +0x52
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start in goroutine 2016071
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/util/wait/wait.go:70 +0x73

goroutine 6972063 [sync.Cond.Wait]:
sync.runtime_notifyListWait(0xc00133f848, 0x1)
        /usr/local/go/src/runtime/sema.go:569 +0x159
sync.(*Cond).Wait(0x2?)
        /usr/local/go/src/sync/cond.go:70 +0x85
golang.org/x/net/http2.(*pipe).Read(0xc00133f830, {0xc000e020d8, 0x4, 0x4})
        /go/pkg/mod/golang.org/x/net@v0.26.0/http2/pipe.go:76 +0xdf
golang.org/x/net/http2.transportResponseBody.Read({0x0?}, {0xc000e020d8?, 0xc0010edce0?, 0x411f65?})
        /go/pkg/mod/golang.org/x/net@v0.26.0/http2/transport.go:2641 +0x65
io.ReadAtLeast({0x7f95840c5818, 0xc00133f800}, {0xc000e020d8, 0x4, 0x4}, 0x4)
        /usr/local/go/src/io/io.go:335 +0x90
k8s.io/apimachinery/pkg/util/framer.(*lengthDelimitedFrameReader).Read(0xc000880918, {0xc0018f7800, 0x400, 0x400})
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/util/framer/framer.go:76 +0x88
k8s.io/apimachinery/pkg/runtime/serializer/streaming.(*decoder).Decode(0xc000bfcb90, 0x0, {0x29bfff0, 0xc000d0e240})
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/runtime/serializer/streaming/streaming.go:77 +0xa3
k8s.io/client-go/rest/watch.(*Decoder).Decode(0xc001652f80)
        /go/pkg/mod/k8s.io/client-go@v0.30.3/rest/watch/decoder.go:49 +0x4b
k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive(0xc000de5c80)
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/watch/streamwatcher.go:105 +0xdb
created by k8s.io/apimachinery/pkg/watch.NewStreamWatcher in goroutine 2015999
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/watch/streamwatcher.go:76 +0x105

goroutine 6952072 [select, 6 minutes]:
golang.org/x/net/http2.(*clientStream).writeRequest(0xc000a6e300, 0xc000c4a480, 0x0)
        /go/pkg/mod/golang.org/x/net@v0.26.0/http2/transport.go:1536 +0xa85
golang.org/x/net/http2.(*clientStream).doRequest(0xc000a6e300, 0x5fa924?, 0xc000eaea20?)
        /go/pkg/mod/golang.org/x/net@v0.26.0/http2/transport.go:1414 +0x56
created by golang.org/x/net/http2.(*ClientConn).roundTrip in goroutine 111
        /go/pkg/mod/golang.org/x/net@v0.26.0/http2/transport.go:1319 +0x3e5

goroutine 2016101 [select, 104 minutes]:
k8s.io/client-go/tools/cache.(*Reflector).startResync(0xc00127b960, 0xc000c6a600, 0xc000bc8f60, 0xc0014c7920)
        /go/pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/reflector.go:377 +0x10f
created by k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch in goroutine 2016113
        /go/pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/reflector.go:364 +0x4ad
<end of log>

```

---

<div class="post-metadata">

**Author:** ![SuperHammer](https://avatars.discourse-cdn.com/v4/letter/s/ccd318/32.png) [@SuperHammer](https://forums.percona.com/u/SuperHammer)\
**Post date:** [July 31, 2025, 1:44pm UTC](https://forums.percona.com/t/backup-retention-not-applied-in-v1-17-0/38904/4 "2025-07-31T13:44:11Z")

</div>

Nevermind this problem, I couldn’t find it with a search but I found this post [Older backups are not getting deleted, with the keep argument specified](https://forums.percona.com/t/older-backups-are-not-getting-deleted-with-the-keep-argument-specified/34424) in the related posts of this thread

Seems to be fixed in 1.18 [Issue navigator - Percona JIRA](https://perconadev.atlassian.net/issues/?selectedIssue=K8SPSMDB-1180&jql=ORDER%20BY%20created%20DESC)

Therefore, closing this as solved 🙂

---

<div class="post-metadata">

**Author:** ![Ivan\_Groenewold](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/ivan_groenewold/32/6299_2.png) [@Ivan\_Groenewold](https://forums.percona.com/u/Ivan_Groenewold)\
**Post date:** [August 4, 2025, 11:04am UTC](https://forums.percona.com/t/backup-retention-not-applied-in-v1-17-0/38904/5 "2025-08-04T11:04:59Z")

</div>

glad to hear you sorted it out