Backup retention not applied in v1.17.0

MongoDB Percona Operator for MongoDB
1

Description:

In an OKD SNO environment, installing the operator and running it works correctly. When making backups to a local minio, backups work fine, but the set retention is not applied, resulting in a large number of old backups.

Steps to Reproduce:

  • Install an OKD SNO cluster
  • Through the OperatorHub section of the web console, install “Percona Distribution for MongoDB Operator” 1.17.0 provided by Percona
  • Follow the installation steps and use the following CRD when required:
apiVersion: psmdb.percona.com/v1
kind: PerconaServerMongoDB
metadata:
  name: prod
  finalizers:
    - percona.com/delete-psmdb-pods-in-order
spec:
  replsets:
    - expose:
        enabled: false
        exposeType: ClusterIP
      resources:
        limits:
          cpu: 300m
          memory: 0.5G
        requests:
          cpu: 300m
          memory: 0.5G
      affinity:
        antiAffinityTopologyKey: none
      arbiter:
        affinity:
          antiAffinityTopologyKey: none
        enabled: false
        size: 0
      nonvoting:
        resources:
          limits:
            cpu: 300m
            memory: 0.5G
          requests:
            cpu: 300m
            memory: 0.5G
        affinity:
          antiAffinityTopologyKey: none
        volumeSpec:
          persistentVolumeClaim:
            resources:
              requests:
                storage: 3Gi
        podDisruptionBudget: {}
        enabled: false
        size: 1
      volumeSpec:
        persistentVolumeClaim:
          resources:
            requests:
              storage: 5Gi
      podDisruptionBudget: {}
      name: rs0
      size: 3
  multiCluster:
    enabled: false
    DNSSuffix: svc.clusterset.local
  upgradeOptions:
    versionServiceEndpoint: 'https://check.percona.com'
    apply: disabled
    schedule: 0 2 * * *
    setFCV: false
  backup:
    pitr:
      enabled: false
      compressionType: gzip
      compressionLevel: 6
    enabled: true
    image: percona/percona-backup-mongodb:2.5.0
    serviceAccountName: percona-server-mongodb-operator
    storages:
      minio:
        type: s3
        s3:
          bucket: pmdb-prod-backup
          region: us-east-1
          credentialsSecret: prod-backup-minio-s3
          endpointUrl: http://minio-api.minio.svc.cluster.local:9000
          insecureSkipTLSVerify: false
    tasks:
      - compressionType: gzip
        enabled: true
        keep: 3
        name: daily-backup
        schedule: 0 0 * * *
        storageName: minio
  secrets:
    users: pmdb-mongodb-prod
    encryptionKey: my-cluster-name-mongodb-encryption-key
  sharding:
    configsvrReplSet:
      expose:
        enabled: false
        exposeType: ClusterIP
      resources:
        limits:
          cpu: 300m
          memory: 0.5G
        requests:
          cpu: 300m
          memory: 0.5G
      affinity:
        antiAffinityTopologyKey: none
      volumeSpec:
        persistentVolumeClaim:
          resources:
            requests:
              storage: 3Gi
      podDisruptionBudget: {}
      size: 1
    mongos:
      expose:
        exposeType: ClusterIP
      resources:
        limits:
          cpu: 300m
          memory: 0.5G
        requests:
          cpu: 300m
          memory: 0.5G
      affinity:
        antiAffinityTopologyKey: none
      podDisruptionBudget: {}
      size: 1
    enabled: false
  pmm:
    enabled: false
    image: percona/pmm-client:2.42.0
    serverHost: monitoring-service
  crVersion: 1.17.0
  image: percona/percona-server-mongodb:7.0.12-7
  imagePullPolicy: Always
  allowUnsafeConfigurations: false
  updateStrategy: SmartUpdate
  • Everything works correctly, the backups are created on a local Minio also installed in the same cluster with a basic configuration
  • Backups retention is not applied after 3 days (.spec.backup.tasks[0].keep)

Version:

Operator version: percona-server-mongodb-operator.v1.17.0 (installed through OperatorHub)

OKD 4.15 version 4.15.0-0.okd-2024-03-10-010116 with kubernetes v1.28.2-3598+6e2789bbd58938-dirty

Logs:

The operator pod has many restarts:

# oc -n prod-pmdb get pods
NAME                                               READY   STATUS    RESTARTS        AGE
percona-server-mongodb-operator-65cd6cfc48-ktwds   1/1     Running   373 (14h ago)   258d
prod-rs0-0                                         2/2     Running   3 (65d ago)     258d
prod-rs0-1                                         2/2     Running   2               258d
prod-rs0-2                                         2/2     Running   3 (84d ago)     258d

The current logs keep repeating the same messages:

<I'll post the logs in a separate message or otherwise the forum won't let me post>

The pod on restart fails with exit code 2:

oc -n prod-pmdb describe pod percona-server-mongodb-operator-65cd6cfc48-ktwds
...
    Command:
      percona-server-mongodb-operator
    State:          Running
      Started:      Wed, 30 Jul 2025 18:07:47 -0400
    Last State:     Terminated
      Reason:       Error
      Exit Code:    2
      Started:      Tue, 29 Jul 2025 10:59:52 -0400
      Finished:     Wed, 30 Jul 2025 18:07:45 -0400
    Ready:          True
    Restart Count:  373
...

These are the last logs for when it was restarted:

<I'll post the logs in a separate message or otherwise the forum won't let me post>

Expected Result:

The retention should be applied and only the last 3 days of backups should be kept.

Actual Result:

The retention is not applied and the backups are not deleted.

Additional Information:

The cluster is running on a single node (OKD SNO), but it shouldn’t matter from an operator perspective.

The CR has a status of “Ready”:

# oc -n prod-pmdb get PerconaServerMongoDB prod
NAME   ENDPOINT                                  STATUS   AGE
prod   prod-rs0.prod-pmdb.svc.cluster.local   ready    258d

Full CR with describe:

# oc -n prod-pmdb describe PerconaServerMongoDB prod
Name:         prod
Namespace:    prod-pmdb
Labels:       <none>
Annotations:  <none>
API Version:  psmdb.percona.com/v1
Kind:         PerconaServerMongoDB
Metadata:
  Creation Timestamp:  2024-11-15T09:06:09Z
  Finalizers:
    percona.com/delete-psmdb-pods-in-order
  Generation:        1
  Resource Version:  95156491
  UID:               ab68cd17-20b3-4e3e-88af-7c12077eb0f3
Spec:
  Allow Unsafe Configurations:  false
  Backup:
    Enabled:  true
    Image:    percona/percona-backup-mongodb:2.5.0
    Pitr:
      Compression Level:   6
      Compression Type:    gzip
      Enabled:             false
    Service Account Name:  percona-server-mongodb-operator
    Storages:
      Minio:
        s3:
          Bucket:                    pmdb-prod-backup
          Credentials Secret:        prod-backup-minio-s3
          Endpoint URL:              http://minio-api.minio.svc.cluster.local:9000
          Insecure Skip TLS Verify:  false
          Region:                    us-east-1
        Type:                        s3
    Tasks:
      Compression Type:  gzip
      Enabled:           true
      Keep:              3
      Name:              daily-backup
      Schedule:          0 0 * * *
      Storage Name:      minio
  Cr Version:            1.17.0
  Image:                 percona/percona-server-mongodb:7.0.12-7
  Image Pull Policy:     Always
  Multi Cluster:
    DNS Suffix:  svc.clusterset.local
    Enabled:     false
  Pmm:
    Enabled:      false
    Image:        percona/pmm-client:2.42.0
    Server Host:  monitoring-service
  Replsets:
    Affinity:
      Anti Affinity Topology Key:  none
    Arbiter:
      Affinity:
        Anti Affinity Topology Key:  none
      Enabled:                       false
      Size:                          0
    Expose:
      Enabled:      false
      Expose Type:  ClusterIP
    Name:           rs0
    Nonvoting:
      Affinity:
        Anti Affinity Topology Key:  none
      Enabled:                       false
      Pod Disruption Budget:
      Resources:
        Limits:
          Cpu:     300m
          Memory:  0.5G
        Requests:
          Cpu:     300m
          Memory:  0.5G
      Size:        1
      Volume Spec:
        Persistent Volume Claim:
          Resources:
            Requests:
              Storage:  3Gi
    Pod Disruption Budget:
    Resources:
      Limits:
        Cpu:     300m
        Memory:  0.5G
      Requests:
        Cpu:     300m
        Memory:  0.5G
    Size:        3
    Volume Spec:
      Persistent Volume Claim:
        Resources:
          Requests:
            Storage:  5Gi
  Secrets:
    Encryption Key:  my-cluster-name-mongodb-encryption-key
    Users:           pmdb-mongodb-prod
  Sharding:
    Configsvr Repl Set:
      Affinity:
        Anti Affinity Topology Key:  none
      Expose:
        Enabled:      false
        Expose Type:  ClusterIP
      Pod Disruption Budget:
      Resources:
        Limits:
          Cpu:     300m
          Memory:  0.5G
        Requests:
          Cpu:     300m
          Memory:  0.5G
      Size:        1
      Volume Spec:
        Persistent Volume Claim:
          Resources:
            Requests:
              Storage:  3Gi
    Enabled:            false
    Mongos:
      Affinity:
        Anti Affinity Topology Key:  none
      Expose:
        Expose Type:  ClusterIP
      Pod Disruption Budget:
      Resources:
        Limits:
          Cpu:     300m
          Memory:  0.5G
        Requests:
          Cpu:      300m
          Memory:   0.5G
      Size:         1
  Update Strategy:  SmartUpdate
  Upgrade Options:
    Apply:                     disabled
    Schedule:                  0 2 * * *
    Set FCV:                   false
    Version Service Endpoint:  https://check.percona.com
Status:
  Conditions:
    Last Transition Time:  2024-11-15T09:06:11Z
    Status:                True
    Type:                  initializing
    Last Transition Time:  2024-11-15T09:07:12Z
    Message:               rs0: ready
    Reason:                RSReady
    Status:                True
    Type:                  ready
    Last Transition Time:  2024-11-15T09:07:12Z
    Status:                True
    Type:                  initializing
    Last Transition Time:  2024-11-15T09:07:23Z
    Status:                True
    Type:                  ready
    Last Transition Time:  2025-05-08T00:00:52Z
    Status:                True
    Type:                  initializing
    Last Transition Time:  2025-05-08T00:01:23Z
    Message:               rs0: ready
    Reason:                RSReady
    Status:                True
    Type:                  ready
    Last Transition Time:  2025-05-08T00:02:36Z
    Status:                True
    Type:                  initializing
    Last Transition Time:  2025-05-08T00:02:47Z
    Message:               rs0: ready
    Reason:                RSReady
    Status:                True
    Type:                  ready
    Last Transition Time:  2025-05-08T00:02:47Z
    Status:                True
    Type:                  initializing
    Last Transition Time:  2025-05-08T00:02:47Z
    Status:                True
    Type:                  ready
    Last Transition Time:  2025-05-27T00:07:34Z
    Status:                True
    Type:                  initializing
    Last Transition Time:  2025-05-27T00:07:50Z
    Message:               rs0: ready
    Reason:                RSReady
    Status:                True
    Type:                  ready
    Last Transition Time:  2025-07-31T13:15:50Z
    Status:                True
    Type:                  initializing
    Last Transition Time:  2025-07-31T13:16:06Z
    Message:               rs0: ready
    Reason:                RSReady
    Status:                True
    Type:                  ready
    Last Transition Time:  2025-07-31T13:16:06Z
    Status:                True
    Type:                  initializing
    Last Transition Time:  2025-07-31T13:16:11Z
    Status:                True
    Type:                  ready
  Host:                    prod-rs0.prod-pmdb.svc.cluster.local
  Mongo Image:             percona/percona-server-mongodb:7.0.12-7
  Mongo Version:           7.0.12-7
  Observed Generation:     1
  Ready:                   3
  Replsets:
    rs0:
      Initialized:  true
      Ready:        3
      Size:         3
      Status:       ready
  Size:             3
  State:            ready
Events:             <none>

The backups are really small as the total database size is <1GB on both disk and DB.

As of now there are more than 200+ backups ready but not deleted:

<I'll post the output in a separate message or otherwise the forum won't let me post>

Here is the full YAML of the oldest backup:

# oc -n prod-pmdb get psmdb-backup cron-prod-20241116000000-wcj8f
NAME                             CLUSTER   STORAGE   DESTINATION                                  TYPE      STATUS   COMPLETED   AGE
cron-prod-20241116000000-wcj8f   prod      minio     s3://pmdb-prod-backup/2024-11-16T00:00:21Z   logical   ready    257d        257d
# oc -n prod-pmdb get psmdb-backup cron-prod-20241116000000-wcj8f -o yaml
apiVersion: psmdb.percona.com/v1
kind: PerconaServerMongoDBBackup
metadata:
  creationTimestamp: "2024-11-16T00:00:00Z"
  finalizers:
  - percona.com/delete-backup
  generateName: cron-prod-20241116000000-
  generation: 1
  labels:
    app.kubernetes.io/instance: prod
    app.kubernetes.io/managed-by: percona-server-mongodb-operator
    app.kubernetes.io/name: percona-server-mongodb
    app.kubernetes.io/part-of: percona-server-mongodb
    percona.com/backup-ancestor: daily-backup
    percona.com/backup-type: cron
    percona.com/cluster: prod
  name: cron-prod-20241116000000-wcj8f
  namespace: prod-pmdb
  resourceVersion: "2570408"
  uid: 6f2f9b3b-8a55-43a4-b39b-f5fc6f7b2b2e
spec:
  clusterName: prod
  compressionType: gzip
  storageName: minio
  type: logical
status:
  completed: "2024-11-16T00:00:31Z"
  destination: s3://pmdb-prod-backup/2024-11-16T00:00:21Z
  lastTransition: "2024-11-16T00:00:31Z"
  pbmName: "2024-11-16T00:00:21Z"
  pbmPod: prod-rs0-0
  pbmPods:
    rs0: prod-rs0-0
  replsetNames:
  - rs0
  s3:
    bucket: pmdb-prod-backup
    credentialsSecret: prod-backup-minio-s3
    endpointUrl: http://minio-api.minio.svc.cluster.local:9000
    region: us-east-1
    serverSideEncryption: {}
  start: "2024-11-16T00:00:21Z"
  state: ready
  storageName: minio
  type: logical

And if I try to delete it, it works correctly:

# oc -n prod-pmdb delete psmdb-backup cron-prod-20241116000000-wcj8f
^C
# oc -n prod-pmdb delete psmdb-backup cron-prod-20241116000000-wcj8f
Error from server (NotFound): perconaservermongodbbackups.psmdb.percona.com "cron-prod-20241116000000-wcj8f" not found

As soon as possible I’d like to replicate this behaviour in a specific lab, but I’m opening an issue in the meantime in case it could be of help and in case anybody could suggest any more specific troubleshooting steps.

2

Backup CRs:

oc -n prod-pmdb get psmdb-backup
NAME                             CLUSTER   STORAGE   DESTINATION                                  TYPE      STATUS   COMPLETED   AGE
...
cron-prod-20241227000000-h4g5j   prod      minio     s3://pmdb-prod-backup/2024-12-27T00:00:21Z   logical   ready    216d        216d
cron-prod-20241228000000-9956b   prod      minio     s3://pmdb-prod-backup/2024-12-28T00:00:21Z   logical   ready    215d        215d
cron-prod-20241229000000-7nxqr   prod      minio     s3://pmdb-prod-backup/2024-12-29T00:00:21Z   logical   ready    214d        214d
cron-prod-20241230000000-m5ck9   prod      minio     s3://pmdb-prod-backup/2024-12-30T00:00:21Z   logical   ready    213d        213d
cron-prod-20241231000000-xzp8n   prod      minio     s3://pmdb-prod-backup/2024-12-31T00:00:21Z   logical   ready    212d        212d
cron-prod-20250101000000-ldkpn   prod      minio     s3://pmdb-prod-backup/2025-01-01T00:00:21Z   logical   ready    211d        211d
cron-prod-20250102000000-5jmdf   prod      minio     s3://pmdb-prod-backup/2025-01-02T00:00:21Z   logical   ready    210d        210d
cron-prod-20250103000000-5s8c6   prod      minio     s3://pmdb-prod-backup/2025-01-03T00:00:21Z   logical   ready    209d        209d
cron-prod-20250104000000-2x2pj   prod      minio     s3://pmdb-prod-backup/2025-01-04T00:00:21Z   logical   ready    208d        208d
cron-prod-20250105000000-k4f2v   prod      minio     s3://pmdb-prod-backup/2025-01-05T00:00:21Z   logical   ready    207d        207d
cron-prod-20250106000000-slcqr   prod      minio     s3://pmdb-prod-backup/2025-01-06T00:00:21Z   logical   ready    206d        206d
cron-prod-20250107000000-fvd2m   prod      minio     s3://pmdb-prod-backup/2025-01-07T00:00:21Z   logical   ready    205d        205d
cron-prod-20250108000000-vpfdk   prod      minio     s3://pmdb-prod-backup/2025-01-08T00:00:21Z   logical   ready    204d        204d
cron-prod-20250109000000-ff9rf   prod      minio     s3://pmdb-prod-backup/2025-01-09T00:00:21Z   logical   ready    203d        203d
cron-prod-20250110000000-swqrl   prod      minio     s3://pmdb-prod-backup/2025-01-10T00:00:21Z   logical   ready    202d        202d
cron-prod-20250111000000-n2lw6   prod      minio     s3://pmdb-prod-backup/2025-01-11T00:00:21Z   logical   ready    201d        201d
cron-prod-20250112000000-g2bdf   prod      minio     s3://pmdb-prod-backup/2025-01-12T00:00:21Z   logical   ready    200d        200d
cron-prod-20250113000000-7r42l   prod      minio     s3://pmdb-prod-backup/2025-01-13T00:00:21Z   logical   ready    199d        199d
cron-prod-20250114000000-nmrh4   prod      minio     s3://pmdb-prod-backup/2025-01-14T00:00:21Z   logical   ready    198d        198d
cron-prod-20250115000000-shn7m   prod      minio     s3://pmdb-prod-backup/2025-01-15T00:00:21Z   logical   ready    197d        197d
cron-prod-20250116000000-zzxtv   prod      minio     s3://pmdb-prod-backup/2025-01-16T00:00:21Z   logical   ready    196d        196d
cron-prod-20250117000000-s5p5b   prod      minio     s3://pmdb-prod-backup/2025-01-17T00:00:21Z   logical   ready    195d        195d
cron-prod-20250118000000-tn82q   prod      minio     s3://pmdb-prod-backup/2025-01-18T00:00:21Z   logical   ready    194d        194d
cron-prod-20250119000000-ns5sn   prod      minio     s3://pmdb-prod-backup/2025-01-19T00:00:21Z   logical   ready    193d        193d
cron-prod-20250120000000-8wcm6   prod      minio     s3://pmdb-prod-backup/2025-01-20T00:00:21Z   logical   ready    192d        192d
cron-prod-20250121000000-jw9mx   prod      minio     s3://pmdb-prod-backup/2025-01-21T00:00:21Z   logical   ready    191d        191d
cron-prod-20250122000000-w6ms7   prod      minio     s3://pmdb-prod-backup/2025-01-22T00:00:21Z   logical   ready    190d        190d
cron-prod-20250123000000-5xn9w   prod      minio     s3://pmdb-prod-backup/2025-01-23T00:00:21Z   logical   ready    189d        189d
cron-prod-20250124000000-cl2jg   prod      minio     s3://pmdb-prod-backup/2025-01-24T00:00:21Z   logical   ready    188d        188d
cron-prod-20250125000000-kndjp   prod      minio     s3://pmdb-prod-backup/2025-01-25T00:00:21Z   logical   ready    187d        187d
cron-prod-20250126000000-ntv87   prod      minio     s3://pmdb-prod-backup/2025-01-26T00:00:21Z   logical   ready    186d        186d
cron-prod-20250127000000-27frf   prod      minio     s3://pmdb-prod-backup/2025-01-27T00:00:21Z   logical   ready    185d        185d
cron-prod-20250128000000-8ghcf   prod      minio     s3://pmdb-prod-backup/2025-01-28T00:00:21Z   logical   ready    184d        184d
cron-prod-20250129000000-p5rns   prod      minio     s3://pmdb-prod-backup/2025-01-29T00:00:21Z   logical   ready    183d        183d
cron-prod-20250130000000-fsxvk   prod      minio     s3://pmdb-prod-backup/2025-01-30T00:00:21Z   logical   ready    182d        182d
cron-prod-20250131000000-cspb5   prod      minio     s3://pmdb-prod-backup/2025-01-31T00:00:21Z   logical   ready    181d        181d
cron-prod-20250201000000-rgz4d   prod      minio     s3://pmdb-prod-backup/2025-02-01T00:00:21Z   logical   ready    180d        180d
cron-prod-20250202000000-wjwlx   prod      minio     s3://pmdb-prod-backup/2025-02-02T00:00:21Z   logical   ready    179d        179d
cron-prod-20250203000000-xhk9h   prod      minio     s3://pmdb-prod-backup/2025-02-03T00:00:21Z   logical   ready    178d        178d
cron-prod-20250204000000-wssnw   prod      minio     s3://pmdb-prod-backup/2025-02-04T00:00:21Z   logical   ready    177d        177d
cron-prod-20250205000000-82hcn   prod      minio     s3://pmdb-prod-backup/2025-02-05T00:00:21Z   logical   ready    176d        176d
cron-prod-20250206000000-85dqc   prod      minio     s3://pmdb-prod-backup/2025-02-06T00:00:21Z   logical   ready    175d        175d
cron-prod-20250207000000-m58tx   prod      minio     s3://pmdb-prod-backup/2025-02-07T00:00:21Z   logical   ready    174d        174d
cron-prod-20250208000000-gntcl   prod      minio     s3://pmdb-prod-backup/2025-02-08T00:00:21Z   logical   ready    173d        173d
cron-prod-20250209000000-7gwlr   prod      minio     s3://pmdb-prod-backup/2025-02-09T00:00:21Z   logical   ready    172d        172d
cron-prod-20250210000000-5z4qk   prod      minio     s3://pmdb-prod-backup/2025-02-10T00:00:21Z   logical   ready    171d        171d
cron-prod-20250211000000-2xz2g   prod      minio     s3://pmdb-prod-backup/2025-02-11T00:00:21Z   logical   ready    170d        170d
cron-prod-20250212000000-dmhwv   prod      minio     s3://pmdb-prod-backup/2025-02-12T00:00:21Z   logical   ready    169d        169d
cron-prod-20250213000000-frtfr   prod      minio     s3://pmdb-prod-backup/2025-02-13T00:00:21Z   logical   ready    168d        168d
cron-prod-20250214000000-fxjf4   prod      minio     s3://pmdb-prod-backup/2025-02-14T00:00:21Z   logical   ready    167d        167d
cron-prod-20250215000000-v7pvp   prod      minio     s3://pmdb-prod-backup/2025-02-15T00:00:21Z   logical   ready    166d        166d
cron-prod-20250216000000-6xbf4   prod      minio     s3://pmdb-prod-backup/2025-02-16T00:00:21Z   logical   ready    165d        165d
cron-prod-20250217000000-mvzxl   prod      minio     s3://pmdb-prod-backup/2025-02-17T00:00:21Z   logical   ready    164d        164d
cron-prod-20250218000000-8pdgx   prod      minio     s3://pmdb-prod-backup/2025-02-18T00:00:21Z   logical   ready    163d        163d
cron-prod-20250219000000-8w4s6   prod      minio     s3://pmdb-prod-backup/2025-02-19T00:00:21Z   logical   ready    162d        162d
cron-prod-20250220000000-blxjx   prod      minio     s3://pmdb-prod-backup/2025-02-20T00:00:21Z   logical   ready    161d        161d
cron-prod-20250221000000-lrhkn   prod      minio     s3://pmdb-prod-backup/2025-02-21T00:00:21Z   logical   ready    160d        160d
cron-prod-20250222000000-kfwwh   prod      minio     s3://pmdb-prod-backup/2025-02-22T00:00:21Z   logical   ready    159d        159d
cron-prod-20250223000000-lcx5w   prod      minio     s3://pmdb-prod-backup/2025-02-23T00:00:21Z   logical   ready    158d        158d
cron-prod-20250224000000-jjxl7   prod      minio     s3://pmdb-prod-backup/2025-02-24T00:00:21Z   logical   ready    157d        157d
cron-prod-20250225000000-qqhw5   prod      minio     s3://pmdb-prod-backup/2025-02-25T00:00:21Z   logical   ready    156d        156d
cron-prod-20250226000000-zrgwj   prod      minio     s3://pmdb-prod-backup/2025-02-26T00:00:21Z   logical   ready    155d        155d
cron-prod-20250227000000-pn2dp   prod      minio     s3://pmdb-prod-backup/2025-02-27T00:00:21Z   logical   ready    154d        154d
cron-prod-20250228000000-w49k5   prod      minio     s3://pmdb-prod-backup/2025-02-28T00:00:21Z   logical   ready    153d        153d
cron-prod-20250301000000-xrmqk   prod      minio     s3://pmdb-prod-backup/2025-03-01T00:00:21Z   logical   ready    152d        152d
cron-prod-20250302000000-npflb   prod      minio     s3://pmdb-prod-backup/2025-03-02T00:00:21Z   logical   ready    151d        151d
cron-prod-20250303000000-gv7mk   prod      minio     s3://pmdb-prod-backup/2025-03-03T00:00:21Z   logical   ready    150d        150d
cron-prod-20250304000000-bsx8x   prod      minio     s3://pmdb-prod-backup/2025-03-04T00:00:21Z   logical   ready    149d        149d
cron-prod-20250305000000-q2lsz   prod      minio     s3://pmdb-prod-backup/2025-03-05T00:00:21Z   logical   ready    148d        148d
cron-prod-20250306000000-r9xgq   prod      minio     s3://pmdb-prod-backup/2025-03-06T00:00:21Z   logical   ready    147d        147d
cron-prod-20250307000000-msx72   prod      minio     s3://pmdb-prod-backup/2025-03-07T00:00:21Z   logical   ready    146d        146d
cron-prod-20250308000000-zgmfj   prod      minio     s3://pmdb-prod-backup/2025-03-08T00:00:21Z   logical   ready    145d        145d
cron-prod-20250309000000-4t4lz   prod      minio     s3://pmdb-prod-backup/2025-03-09T00:00:21Z   logical   ready    144d        144d
cron-prod-20250310000000-qgblc   prod      minio     s3://pmdb-prod-backup/2025-03-10T00:00:21Z   logical   ready    143d        143d
cron-prod-20250311000000-5wjhf   prod      minio     s3://pmdb-prod-backup/2025-03-11T00:00:21Z   logical   ready    142d        142d
cron-prod-20250312000000-4slw7   prod      minio     s3://pmdb-prod-backup/2025-03-12T00:00:21Z   logical   ready    141d        141d
cron-prod-20250313000000-9fjjf   prod      minio     s3://pmdb-prod-backup/2025-03-13T00:00:21Z   logical   ready    140d        140d
cron-prod-20250314000000-pcw79   prod      minio     s3://pmdb-prod-backup/2025-03-14T00:00:21Z   logical   ready    139d        139d
cron-prod-20250315000000-bjlsq   prod      minio     s3://pmdb-prod-backup/2025-03-15T00:00:21Z   logical   ready    138d        138d
cron-prod-20250316000000-pkqhm   prod      minio     s3://pmdb-prod-backup/2025-03-16T00:00:21Z   logical   ready    137d        137d
cron-prod-20250317000000-l7g5m   prod      minio     s3://pmdb-prod-backup/2025-03-17T00:00:21Z   logical   ready    136d        136d
cron-prod-20250318000000-sgqrl   prod      minio     s3://pmdb-prod-backup/2025-03-18T00:00:21Z   logical   ready    135d        135d
cron-prod-20250319000000-5hwkm   prod      minio     s3://pmdb-prod-backup/2025-03-19T00:00:21Z   logical   ready    134d        134d
cron-prod-20250320000000-nmtcv   prod      minio     s3://pmdb-prod-backup/2025-03-20T00:00:21Z   logical   ready    133d        133d
cron-prod-20250321000000-rgx85   prod      minio     s3://pmdb-prod-backup/2025-03-21T00:00:21Z   logical   ready    132d        132d
cron-prod-20250322000000-fr966   prod      minio     s3://pmdb-prod-backup/2025-03-22T00:00:21Z   logical   ready    131d        131d
cron-prod-20250323000000-htlmd   prod      minio     s3://pmdb-prod-backup/2025-03-23T00:00:21Z   logical   ready    130d        130d
cron-prod-20250324000000-sd2hx   prod      minio     s3://pmdb-prod-backup/2025-03-24T00:00:21Z   logical   ready    129d        129d
cron-prod-20250325000000-tnf8b   prod      minio     s3://pmdb-prod-backup/2025-03-25T00:00:21Z   logical   ready    128d        128d
...
cron-prod-20250510000000-7pqc4   prod      minio     s3://pmdb-prod-backup/2025-05-10T00:00:21Z   logical   ready    82d         82d
cron-prod-20250511000000-qkzcc   prod      minio     s3://pmdb-prod-backup/2025-05-11T00:00:21Z   logical   ready    81d         81d
cron-prod-20250512000000-gccsn   prod      minio     s3://pmdb-prod-backup/2025-05-12T00:00:21Z   logical   ready    80d         80d
cron-prod-20250513000000-sscsn   prod      minio     s3://pmdb-prod-backup/2025-05-13T00:00:21Z   logical   ready    79d         79d
cron-prod-20250514000000-4m6qn   prod      minio     s3://pmdb-prod-backup/2025-05-14T00:00:21Z   logical   ready    78d         78d
cron-prod-20250515000000-r6lj5   prod      minio     s3://pmdb-prod-backup/2025-05-15T00:00:21Z   logical   ready    77d         77d
cron-prod-20250516000000-24m7n   prod      minio     s3://pmdb-prod-backup/2025-05-16T00:00:21Z   logical   ready    76d         76d
cron-prod-20250517000000-bk2lj   prod      minio     s3://pmdb-prod-backup/2025-05-17T00:00:21Z   logical   ready    75d         75d
cron-prod-20250518000000-99g9s   prod      minio     s3://pmdb-prod-backup/2025-05-18T00:00:21Z   logical   ready    74d         74d
cron-prod-20250519000000-wsccr   prod      minio     s3://pmdb-prod-backup/2025-05-19T00:00:21Z   logical   ready    73d         73d
cron-prod-20250520000000-c4fkc   prod      minio     s3://pmdb-prod-backup/2025-05-20T00:00:21Z   logical   ready    72d         72d
cron-prod-20250521000000-slmvl   prod      minio     s3://pmdb-prod-backup/2025-05-21T00:00:21Z   logical   ready    71d         71d
cron-prod-20250522000000-hr5tx   prod      minio     s3://pmdb-prod-backup/2025-05-22T00:00:21Z   logical   ready    70d         70d
cron-prod-20250523000000-k462p   prod      minio     s3://pmdb-prod-backup/2025-05-23T00:00:21Z   logical   ready    69d         69d
cron-prod-20250524000000-tv6cj   prod      minio     s3://pmdb-prod-backup/2025-05-24T00:00:21Z   logical   ready    68d         68d
cron-prod-20250525000000-8vgt8   prod      minio     s3://pmdb-prod-backup/2025-05-25T00:00:21Z   logical   ready    67d         67d
cron-prod-20250526000000-4hgcn   prod      minio     s3://pmdb-prod-backup/2025-05-26T00:00:21Z   logical   ready    66d         66d
cron-prod-20250527000000-56j4k   prod      minio     s3://pmdb-prod-backup/2025-05-27T00:00:21Z   logical   ready    65d         65d
cron-prod-20250528000000-vhpww   prod      minio     s3://pmdb-prod-backup/2025-05-28T00:00:21Z   logical   ready    64d         64d
cron-prod-20250529000000-2w2k5   prod      minio     s3://pmdb-prod-backup/2025-05-29T00:00:21Z   logical   ready    63d         63d
cron-prod-20250530000000-s6d7c   prod      minio     s3://pmdb-prod-backup/2025-05-30T00:00:21Z   logical   ready    62d         62d
cron-prod-20250531000000-28gbz   prod      minio     s3://pmdb-prod-backup/2025-05-31T00:00:21Z   logical   ready    61d         61d
cron-prod-20250601000000-txtxs   prod      minio     s3://pmdb-prod-backup/2025-06-01T00:00:21Z   logical   ready    60d         60d
cron-prod-20250602000000-86dtz   prod      minio     s3://pmdb-prod-backup/2025-06-02T00:00:21Z   logical   ready    59d         59d
cron-prod-20250603000000-f27gx   prod      minio     s3://pmdb-prod-backup/2025-06-03T00:00:21Z   logical   ready    58d         58d
cron-prod-20250604000000-hhs6n   prod      minio     s3://pmdb-prod-backup/2025-06-04T00:00:21Z   logical   ready    57d         57d
cron-prod-20250605000000-ll45p   prod      minio     s3://pmdb-prod-backup/2025-06-05T00:00:21Z   logical   ready    56d         56d
cron-prod-20250606000000-f57lz   prod      minio     s3://pmdb-prod-backup/2025-06-06T00:00:21Z   logical   ready    55d         55d
cron-prod-20250607000000-f9fss   prod      minio     s3://pmdb-prod-backup/2025-06-07T00:00:21Z   logical   ready    54d         54d
cron-prod-20250608000000-z22zs   prod      minio     s3://pmdb-prod-backup/2025-06-08T00:00:21Z   logical   ready    53d         53d
cron-prod-20250609000000-xpgmg   prod      minio     s3://pmdb-prod-backup/2025-06-09T00:00:21Z   logical   ready    52d         52d
cron-prod-20250610000000-k95q6   prod      minio     s3://pmdb-prod-backup/2025-06-10T00:00:21Z   logical   ready    51d         51d
cron-prod-20250611000000-k6bgm   prod      minio     s3://pmdb-prod-backup/2025-06-11T00:00:21Z   logical   ready    50d         50d
cron-prod-20250612000000-s5gv4   prod      minio     s3://pmdb-prod-backup/2025-06-12T00:00:21Z   logical   ready    49d         49d
cron-prod-20250613000000-2jzq6   prod      minio     s3://pmdb-prod-backup/2025-06-13T00:00:21Z   logical   ready    48d         48d
cron-prod-20250614000000-jkmrd   prod      minio     s3://pmdb-prod-backup/2025-06-14T00:00:21Z   logical   ready    47d         47d
cron-prod-20250615000000-h9kg8   prod      minio     s3://pmdb-prod-backup/2025-06-15T00:00:21Z   logical   ready    46d         46d
cron-prod-20250616000000-fqp48   prod      minio     s3://pmdb-prod-backup/2025-06-16T00:00:21Z   logical   ready    45d         45d
cron-prod-20250617000000-64tmx   prod      minio     s3://pmdb-prod-backup/2025-06-17T00:00:21Z   logical   ready    44d         44d
cron-prod-20250618000000-mmk5v   prod      minio     s3://pmdb-prod-backup/2025-06-18T00:00:21Z   logical   ready    43d         43d
cron-prod-20250619000000-jckw7   prod      minio     s3://pmdb-prod-backup/2025-06-19T00:00:21Z   logical   ready    42d         42d
cron-prod-20250620000000-qp7ps   prod      minio     s3://pmdb-prod-backup/2025-06-20T00:00:21Z   logical   ready    41d         41d
cron-prod-20250621000000-2qpxd   prod      minio     s3://pmdb-prod-backup/2025-06-21T00:00:21Z   logical   ready    40d         40d
cron-prod-20250622000000-n9fx5   prod      minio     s3://pmdb-prod-backup/2025-06-22T00:00:21Z   logical   ready    39d         39d
cron-prod-20250623000000-kbj7t   prod      minio     s3://pmdb-prod-backup/2025-06-23T00:00:21Z   logical   ready    38d         38d
cron-prod-20250624000000-9db66   prod      minio     s3://pmdb-prod-backup/2025-06-24T00:00:21Z   logical   ready    37d         37d
cron-prod-20250625000000-2qwkg   prod      minio     s3://pmdb-prod-backup/2025-06-25T00:00:21Z   logical   ready    36d         36d
cron-prod-20250626000000-fz22n   prod      minio     s3://pmdb-prod-backup/2025-06-26T00:00:21Z   logical   ready    35d         35d
cron-prod-20250627000000-rbffp   prod      minio     s3://pmdb-prod-backup/2025-06-27T00:00:21Z   logical   ready    34d         34d
cron-prod-20250628000000-slldt   prod      minio     s3://pmdb-prod-backup/2025-06-28T00:00:21Z   logical   ready    33d         33d
cron-prod-20250629000000-5qqdc   prod      minio     s3://pmdb-prod-backup/2025-06-29T00:00:21Z   logical   ready    32d         32d
cron-prod-20250630000000-g2c5b   prod      minio     s3://pmdb-prod-backup/2025-06-30T00:00:21Z   logical   ready    31d         31d
cron-prod-20250701000000-k82lh   prod      minio     s3://pmdb-prod-backup/2025-07-01T00:00:21Z   logical   ready    30d         30d
cron-prod-20250702000000-65h4k   prod      minio     s3://pmdb-prod-backup/2025-07-02T00:00:21Z   logical   ready    29d         29d
cron-prod-20250703000000-zm799   prod      minio     s3://pmdb-prod-backup/2025-07-03T00:00:21Z   logical   ready    28d         28d
cron-prod-20250704000000-jxkbm   prod      minio     s3://pmdb-prod-backup/2025-07-04T00:00:21Z   logical   ready    27d         27d
cron-prod-20250705000000-bxh27   prod      minio     s3://pmdb-prod-backup/2025-07-05T00:00:21Z   logical   ready    26d         26d
cron-prod-20250706000000-dtrsk   prod      minio     s3://pmdb-prod-backup/2025-07-06T00:00:21Z   logical   ready    25d         25d
cron-prod-20250707000000-rq72q   prod      minio     s3://pmdb-prod-backup/2025-07-07T00:00:21Z   logical   ready    24d         24d
cron-prod-20250708000000-xf5pm   prod      minio     s3://pmdb-prod-backup/2025-07-08T00:00:21Z   logical   ready    23d         23d
cron-prod-20250709000000-8lwgw   prod      minio     s3://pmdb-prod-backup/2025-07-09T00:00:21Z   logical   ready    22d         22d
cron-prod-20250710000000-vrn7x   prod      minio     s3://pmdb-prod-backup/2025-07-10T00:00:21Z   logical   ready    21d         21d
cron-prod-20250711000000-h9g5c   prod      minio     s3://pmdb-prod-backup/2025-07-11T00:00:21Z   logical   ready    20d         20d
cron-prod-20250712000000-6fsnt   prod      minio     s3://pmdb-prod-backup/2025-07-12T00:00:21Z   logical   ready    19d         19d
cron-prod-20250713000000-nftwc   prod      minio     s3://pmdb-prod-backup/2025-07-13T00:00:21Z   logical   ready    18d         18d
cron-prod-20250714000000-s2x7x   prod      minio     s3://pmdb-prod-backup/2025-07-14T00:00:21Z   logical   ready    17d         17d
cron-prod-20250715000000-xkbmc   prod      minio     s3://pmdb-prod-backup/2025-07-15T00:00:21Z   logical   ready    16d         16d
cron-prod-20250716000000-9qx9w   prod      minio     s3://pmdb-prod-backup/2025-07-16T00:00:21Z   logical   ready    15d         15d
cron-prod-20250717000000-7l264   prod      minio     s3://pmdb-prod-backup/2025-07-17T00:00:21Z   logical   ready    14d         14d
cron-prod-20250718000000-xb6pk   prod      minio     s3://pmdb-prod-backup/2025-07-18T00:00:21Z   logical   ready    13d         13d
cron-prod-20250719000000-zqh4m   prod      minio     s3://pmdb-prod-backup/2025-07-19T00:00:21Z   logical   ready    12d         12d
cron-prod-20250720000000-lwnp4   prod      minio     s3://pmdb-prod-backup/2025-07-20T00:00:21Z   logical   ready    11d         11d
cron-prod-20250721000000-kdrjh   prod      minio     s3://pmdb-prod-backup/2025-07-21T00:00:21Z   logical   ready    10d         10d
cron-prod-20250722000000-xg8x9   prod      minio     s3://pmdb-prod-backup/2025-07-22T00:00:21Z   logical   ready    9d          9d
cron-prod-20250723000000-fhxn2   prod      minio     s3://pmdb-prod-backup/2025-07-23T00:00:21Z   logical   ready    8d          8d
cron-prod-20250724000000-9n6x6   prod      minio     s3://pmdb-prod-backup/2025-07-24T00:00:21Z   logical   ready    7d13h       7d13h
cron-prod-20250725000000-gpg5f   prod      minio     s3://pmdb-prod-backup/2025-07-25T00:00:21Z   logical   ready    6d13h       6d13h
cron-prod-20250726000000-lxvg6   prod      minio     s3://pmdb-prod-backup/2025-07-26T00:00:21Z   logical   ready    5d13h       5d13h
cron-prod-20250727000000-7n9s9   prod      minio     s3://pmdb-prod-backup/2025-07-27T00:00:21Z   logical   ready    4d13h       4d13h
...

Current logs:

oc -n prod-pmdb logs percona-server-mongodb-operator-65cd6cfc48-ktwds
2025-07-30T22:07:47.605Z        INFO    setup   Manager starting up     {"gitCommit": "5019408f1fe40483fc5effaf61ab3f672765b189", "gitBranch": "release-1-17-0", "goVersion": "go1.22.6", "os": "linux", "arch": "amd64"}
2025-07-30T22:07:47.616Z        INFO    server version  {"platform": "openshift", "version": "undefined (v4.0+)"}
2025-07-30T22:07:47.631Z        INFO    controller-runtime.metrics      Starting metrics server
2025-07-30T22:07:47.631Z        INFO    starting server {"name": "health probe", "addr": "[::]:8081"}
2025-07-30T22:07:47.631Z        INFO    controller-runtime.metrics      Serving metrics server  {"bindAddress": ":8080", "secure": false}
I0730 22:07:47.631676       1 leaderelection.go:250] attempting to acquire leader lease prod-pmdb/08db0feb.percona.com...
I0730 22:08:03.851549       1 leaderelection.go:260] successfully acquired lease prod-pmdb/08db0feb.percona.com
2025-07-30T22:08:03.851Z        INFO    Starting EventSource    {"controller": "psmdbrestore-controller", "source": "kind source: *v1.PerconaServerMongoDBRestore"}
2025-07-30T22:08:03.851Z        INFO    Starting EventSource    {"controller": "psmdbrestore-controller", "source": "kind source: *v1.Pod"}
2025-07-30T22:08:03.851Z        INFO    Starting Controller     {"controller": "psmdbrestore-controller"}
2025-07-30T22:08:03.851Z        INFO    Starting EventSource    {"controller": "psmdbbackup-controller", "source": "kind source: *v1.PerconaServerMongoDBBackup"}
2025-07-30T22:08:03.851Z        INFO    Starting EventSource    {"controller": "psmdbbackup-controller", "source": "kind source: *v1.Pod"}
2025-07-30T22:08:03.851Z        INFO    Starting Controller     {"controller": "psmdbbackup-controller"}
2025-07-30T22:08:03.851Z        INFO    Starting EventSource    {"controller": "psmdb-controller", "source": "kind source: *v1.PerconaServerMongoDB"}
2025-07-30T22:08:03.851Z        INFO    Starting Controller     {"controller": "psmdb-controller"}
2025-07-30T22:08:03.971Z        INFO    Starting workers        {"controller": "psmdbrestore-controller", "worker count": 1}
2025-07-30T22:08:03.972Z        INFO    Starting workers        {"controller": "psmdb-controller", "worker count": 1}
2025-07-30T22:08:03.972Z        INFO    Starting workers        {"controller": "psmdbbackup-controller", "worker count": 1}
2025-07-30T22:08:03.999Z        INFO    Creating or updating backup job {"controller": "psmdb-controller", "object": {"name":"prod","namespace":"prod-pmdb"}, "namespace": "prod-pmdb", "name": "prod", "reconcileID": "83d3f4ff-b856-4965-b8f3-4cdd0b00f5c9", "name": "daily-backup", "namespace": "prod-pmdb", "schedule": "0 0 * * *"}
2025-07-30T22:08:04.191Z        INFO    add new job     {"controller": "psmdb-controller", "object": {"name":"prod","namespace":"prod-pmdb"}, "namespace": "prod-pmdb", "name": "prod", "reconcileID": "83d3f4ff-b856-4965-b8f3-4cdd0b00f5c9", "name": "ensure-version/prod-pmdb/prod", "schedule": "0 2 * * *"}
2025-07-31T00:00:10.285Z        INFO    Starting backup {"controller": "psmdbbackup-controller", "object": {"name":"cron-prod-20250731000000-r6hs6","namespace":"prod-pmdb"}, "namespace": "prod-pmdb", "name": "cron-prod-20250731000000-r6hs6", "reconcileID": "48ad84f1-4b9d-4278-8fe0-88ffbc0453fc", "backup": "cron-prod-20250731000000-r6hs6", "storage": "minio"}
2025-07-31T00:00:10.285Z        INFO    Setting PBM config      {"controller": "psmdbbackup-controller", "object": {"name":"cron-prod-20250731000000-r6hs6","namespace":"prod-pmdb"}, "namespace": "prod-pmdb", "name": "cron-prod-20250731000000-r6hs6", "reconcileID": "48ad84f1-4b9d-4278-8fe0-88ffbc0453fc", "backup": "prod"}
2025-07-31T00:00:21.292Z        INFO    Sending backup command  {"controller": "psmdbbackup-controller", "object": {"name":"cron-prod-20250731000000-r6hs6","namespace":"prod-pmdb"}, "namespace": "prod-pmdb", "name": "cron-prod-20250731000000-r6hs6", "reconcileID": "48ad84f1-4b9d-4278-8fe0-88ffbc0453fc", "backupCmd": "backup [name: 2025-07-31T00:00:21Z, compression: gzip (level: default)] <ts: 0>"}
2025-07-31T13:15:50.875Z        INFO    Cluster state changed   {"controller": "psmdb-controller", "object": {"name":"prod","namespace":"prod-pmdb"}, "namespace": "prod-pmdb", "name": "prod", "reconcileID": "3ff8ee71-81ee-4134-88b8-bb89c9e88a7b", "previous": "ready", "current": "initializing"}
2025-07-31T13:16:11.874Z        INFO    Cluster state changed   {"controller": "psmdb-controller", "object": {"name":"prod","namespace":"prod-pmdb"}, "namespace": "prod-pmdb", "name": "prod", "reconcileID": "8517fb39-cd33-430b-924d-74e92e43e7e7", "previous": "initializing", "current": "ready"}
2025-07-31T13:24:39.943Z        INFO    Setting PBM config      {"controller": "psmdbbackup-controller", "object": {"name":"cron-prod-20241116000000-wcj8f","namespace":"prod-pmdb"}, "namespace": "prod-pmdb", "name": "cron-prod-20241116000000-wcj8f", "reconcileID": "8f558153-0421-45a0-bb96-fb28032e082d", "backup": "prod"}
2025-07-31T13:24:51.004Z        INFO    nothing to delete       {"controller": "psmdbbackup-controller", "object": {"name":"cron-prod-20241116000000-wcj8f","namespace":"prod-pmdb"}, "namespace": "prod-pmdb", "name": "cron-prod-20241116000000-wcj8f", "reconcileID": "8f558153-0421-45a0-bb96-fb28032e082d"}
3

Logs before restart:

oc -n prod-pmdb logs --previous percona-server-mongodb-operator-65cd6cfc48-ktwds
goroutine 191 [select, 1867 minutes]:
reflect.rselect({0xc000e0df00, 0x3, 0x0?})
        /usr/local/go/src/runtime/select.go:589 +0x2c5
reflect.Select({0xc0008c2420?, 0x3, 0xc000cab7d0?})
        /usr/local/go/src/reflect/value.go:3169 +0x5ca
sigs.k8s.io/controller-runtime/pkg/internal/syncs.MergeChans[...].func2()
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/syncs/syncs.go:34 +0x55
created by sigs.k8s.io/controller-runtime/pkg/internal/syncs.MergeChans[...] in goroutine 181
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/syncs/syncs.go:32 +0x438

goroutine 193 [chan receive, 1867 minutes]:
k8s.io/client-go/tools/cache.(*sharedProcessor).run(0xc00088e640, 0xc000bc91a0)
        /go/pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/shared_informer.go:807 +0x4d
k8s.io/client-go/tools/cache.(*sharedIndexInformer).Run.(*Group).StartWithChannel.func4()
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/util/wait/wait.go:55 +0x1b
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/util/wait/wait.go:72 +0x52
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start in goroutine 181
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/util/wait/wait.go:70 +0x73

goroutine 194 [chan receive, 1867 minutes]:
k8s.io/client-go/tools/cache.(*controller).Run.func1()
        /go/pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/controller.go:132 +0x25
created by k8s.io/client-go/tools/cache.(*controller).Run in goroutine 181
        /go/pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/controller.go:131 +0xa9

goroutine 195 [select]:
k8s.io/client-go/tools/cache.watchHandler({0x0?, 0x0?, 0x3b5cf60?}, {0x29c0bd0, 0xc0003eda80}, {0x7f955fd08b28, 0xc0008c24d0}, {0x29f1138, 0x21ea180}, 0x0, ...)
        /go/pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/reflector.go:732 +0x187
k8s.io/client-go/tools/cache.(*Reflector).watch(0xc0008c4380, {0x0?, 0x0?}, 0xc000bc90e0, 0xc000bd4a20)
        /go/pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/reflector.go:445 +0x545
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch(0xc0008c4380, 0xc000bc90e0)
        /go/pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/reflector.go:365 +0x4d7
k8s.io/client-go/tools/cache.(*Reflector).Run.func1()
        /go/pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/reflector.go:298 +0x25
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x10?)
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/util/wait/backoff.go:226 +0x33
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc0005ddf50, {0x29bb420, 0xc00088e820}, 0x1, 0xc000bc90e0)
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/util/wait/backoff.go:227 +0xaf
k8s.io/client-go/tools/cache.(*Reflector).Run(0xc0008c4380, 0xc000bc90e0)
        /go/pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/reflector.go:297 +0x1c5
k8s.io/client-go/tools/cache.(*controller).Run.(*Group).StartWithChannel.func2()
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/util/wait/wait.go:55 +0x1b
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/util/wait/wait.go:72 +0x52
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start in goroutine 181
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/util/wait/wait.go:70 +0x73

goroutine 73 [select, 16 minutes]:
k8s.io/client-go/tools/cache.(*Reflector).startResync(0xc0008c4380, 0xc000bc90e0, 0xc0000b56e0, 0xc000bd4a20)
        /go/pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/reflector.go:377 +0x10f
created by k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch in goroutine 195
        /go/pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/reflector.go:364 +0x4ad

goroutine 172 [select, 1867 minutes]:
reflect.rselect({0xc000eaaf00, 0x3, 0x0?})
        /usr/local/go/src/runtime/select.go:589 +0x2c5
reflect.Select({0xc000c27b80?, 0x3, 0xc000ca7fd0?})
        /usr/local/go/src/reflect/value.go:3169 +0x5ca
sigs.k8s.io/controller-runtime/pkg/internal/syncs.MergeChans[...].func2()
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/syncs/syncs.go:34 +0x55
created by sigs.k8s.io/controller-runtime/pkg/internal/syncs.MergeChans[...] in goroutine 169
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/syncs/syncs.go:32 +0x438

goroutine 174 [chan receive, 1867 minutes]:
k8s.io/client-go/tools/cache.(*sharedProcessor).run(0xc000bfd360, 0xc00073b020)
        /go/pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/shared_informer.go:807 +0x4d
k8s.io/client-go/tools/cache.(*sharedIndexInformer).Run.(*Group).StartWithChannel.func4()
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/util/wait/wait.go:55 +0x1b
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/util/wait/wait.go:72 +0x52
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start in goroutine 169
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/util/wait/wait.go:70 +0x73

goroutine 175 [chan receive, 1867 minutes]:
k8s.io/client-go/tools/cache.(*controller).Run.func1()
        /go/pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/controller.go:132 +0x25
created by k8s.io/client-go/tools/cache.(*controller).Run in goroutine 169
        /go/pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/controller.go:131 +0xa9

goroutine 176 [select]:
k8s.io/client-go/tools/cache.watchHandler({0x0?, 0x0?, 0x3b5cf60?}, {0x29c0bd0, 0xc000905040}, {0x7f955fd08b28, 0xc000c27c30}, {0x29f1138, 0x21f6980}, 0x0, ...)
        /go/pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/reflector.go:732 +0x187
k8s.io/client-go/tools/cache.(*Reflector).watch(0xc0007da380, {0x0?, 0x0?}, 0xc00073af60, 0xc000563680)
        /go/pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/reflector.go:445 +0x545
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch(0xc0007da380, 0xc00073af60)
        /go/pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/reflector.go:365 +0x4d7
k8s.io/client-go/tools/cache.(*Reflector).Run.func1()
        /go/pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/reflector.go:298 +0x25
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x10?)
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/util/wait/backoff.go:226 +0x33
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc0005e1f50, {0x29bb420, 0xc000bfd4a0}, 0x1, 0xc00073af60)
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/util/wait/backoff.go:227 +0xaf
k8s.io/client-go/tools/cache.(*Reflector).Run(0xc0007da380, 0xc00073af60)
        /go/pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/reflector.go:297 +0x1c5
k8s.io/client-go/tools/cache.(*controller).Run.(*Group).StartWithChannel.func2()
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/util/wait/wait.go:55 +0x1b
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/util/wait/wait.go:72 +0x52
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start in goroutine 169
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/util/wait/wait.go:70 +0x73

goroutine 211 [select, 153 minutes]:
k8s.io/client-go/tools/cache.(*Reflector).startResync(0xc0007da380, 0xc00073af60, 0xc00073b3e0, 0xc000563680)
        /go/pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/reflector.go:377 +0x10f
created by k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch in goroutine 176
        /go/pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/reflector.go:364 +0x4ad

goroutine 231 [chan receive, 153 minutes]:
k8s.io/client-go/tools/cache.(*processorListener).run.func1()
        /go/pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/shared_informer.go:973 +0x4b
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/util/wait/backoff.go:226 +0x33
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000127f70, {0x29bb400, 0xc00120c0f0}, 0x1, 0xc0000b4f00)
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/util/wait/backoff.go:227 +0xaf
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000ca7f70, 0x3b9aca00, 0x0, 0x1, 0xc0000b4f00)
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/util/wait/backoff.go:204 +0x7f
k8s.io/apimachinery/pkg/util/wait.Until(...)
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/util/wait/backoff.go:161
k8s.io/client-go/tools/cache.(*processorListener).run(0xc000962000)
        /go/pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/shared_informer.go:972 +0x69
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/util/wait/wait.go:72 +0x52
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start in goroutine 136
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/util/wait/wait.go:70 +0x73

goroutine 232 [select, 153 minutes]:
k8s.io/client-go/tools/cache.(*processorListener).pop(0xc000962000)
        /go/pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/shared_informer.go:943 +0x107
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/util/wait/wait.go:72 +0x52
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start in goroutine 136
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/util/wait/wait.go:70 +0x73

<similar logs omitted due to post size>

goroutine 2016071 [sync.Cond.Wait, 1327 minutes]:
sync.runtime_notifyListWait(0xc000c26238, 0x1)
        /usr/local/go/src/runtime/sema.go:569 +0x159
sync.(*Cond).Wait(0xc000537a00?)
        /usr/local/go/src/sync/cond.go:70 +0x85
k8s.io/client-go/tools/cache.(*DeltaFIFO).Pop(0xc000c26210, 0xc000223eb0)
        /go/pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/delta_fifo.go:575 +0x236
k8s.io/client-go/tools/cache.(*controller).processLoop(0xc000636460)
        /go/pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/controller.go:188 +0x30
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/util/wait/backoff.go:226 +0x33
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000e42df0, {0x29bb400, 0xc000e48870}, 0x1, 0xc000c6a600)
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/util/wait/backoff.go:227 +0xaf
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000e42df0, 0x3b9aca00, 0x0, 0x1, 0xc000c6a600)
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/util/wait/backoff.go:204 +0x7f
k8s.io/apimachinery/pkg/util/wait.Until(...)
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/util/wait/backoff.go:161
k8s.io/client-go/tools/cache.(*controller).Run(0xc000636460, 0xc000c6a600)
        /go/pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/controller.go:159 +0x35e
k8s.io/client-go/tools/cache.(*sharedIndexInformer).Run(0xc001068790, 0xc000c6a600)
        /go/pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/shared_informer.go:508 +0x2c8
sigs.k8s.io/controller-runtime/pkg/cache/internal.(*Cache).Start(0xc000290690, 0xc00163d7d0?)
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/cache/internal/informers.go:106 +0x72
sigs.k8s.io/controller-runtime/pkg/cache/internal.(*Informers).startInformerLocked.func1()
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/cache/internal/informers.go:239 +0x7b
created by sigs.k8s.io/controller-runtime/pkg/cache/internal.(*Informers).startInformerLocked in goroutine 144
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/cache/internal/informers.go:237 +0x87

goroutine 2015993 [sync.Cond.Wait, 1327 minutes]:
sync.runtime_notifyListWait(0xc000b7e398, 0x1)
        /usr/local/go/src/runtime/sema.go:569 +0x159
sync.(*Cond).Wait(0xc0005372a0?)
        /usr/local/go/src/sync/cond.go:70 +0x85
k8s.io/client-go/tools/cache.(*DeltaFIFO).Pop(0xc000b7e370, 0xc00071e5d0)
        /go/pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/delta_fifo.go:575 +0x236
k8s.io/client-go/tools/cache.(*controller).processLoop(0xc000724280)
        /go/pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/controller.go:188 +0x30
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/util/wait/backoff.go:226 +0x33
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000128df0, {0x29bb400, 0xc000a862a0}, 0x1, 0xc000bc82a0)
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/util/wait/backoff.go:227 +0xaf
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000128df0, 0x3b9aca00, 0x0, 0x1, 0xc000bc82a0)
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/util/wait/backoff.go:204 +0x7f
k8s.io/apimachinery/pkg/util/wait.Until(...)
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/util/wait/backoff.go:161
k8s.io/client-go/tools/cache.(*controller).Run(0xc000724280, 0xc000bc82a0)
        /go/pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/controller.go:159 +0x35e
k8s.io/client-go/tools/cache.(*sharedIndexInformer).Run(0xc000b7e000, 0xc000bc82a0)
        /go/pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/shared_informer.go:508 +0x2c8
sigs.k8s.io/controller-runtime/pkg/cache/internal.(*Cache).Start(0xc0006147e0, 0xc000ca9fd0?)
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/cache/internal/informers.go:106 +0x72
sigs.k8s.io/controller-runtime/pkg/cache/internal.(*Informers).startInformerLocked.func1()
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/cache/internal/informers.go:239 +0x7b
created by sigs.k8s.io/controller-runtime/pkg/cache/internal.(*Informers).startInformerLocked in goroutine 144
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/cache/internal/informers.go:237 +0x87

goroutine 2015998 [chan receive, 1327 minutes]:
k8s.io/client-go/tools/cache.(*controller).Run.func1()
        /go/pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/controller.go:132 +0x25
created by k8s.io/client-go/tools/cache.(*controller).Run in goroutine 2015993
        /go/pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/controller.go:131 +0xa9

goroutine 2015995 [select, 1327 minutes]:
reflect.rselect({0xc001031f00, 0x3, 0xc001031e90?})
        /usr/local/go/src/runtime/select.go:589 +0x2c5
reflect.Select({0xc000b7e2c0?, 0x3, 0xc001031fd0?})
        /usr/local/go/src/reflect/value.go:3169 +0x5ca
sigs.k8s.io/controller-runtime/pkg/internal/syncs.MergeChans[...].func2()
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/syncs/syncs.go:34 +0x55
created by sigs.k8s.io/controller-runtime/pkg/internal/syncs.MergeChans[...] in goroutine 2015993
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/syncs/syncs.go:32 +0x438

goroutine 2016098 [select, 65 minutes]:
k8s.io/client-go/tools/cache.(*Reflector).startResync(0xc0015b68c0, 0xc000bc82a0, 0xc000bc8780, 0xc0014c7080)
        /go/pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/reflector.go:377 +0x10f
created by k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch in goroutine 2015999
        /go/pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/reflector.go:364 +0x4ad

goroutine 6955599 [sync.Cond.Wait]:
sync.runtime_notifyListWait(0xc000b4ac48, 0x5)
        /usr/local/go/src/runtime/sema.go:569 +0x159
sync.(*Cond).Wait(0xc000bb0dc0?)
        /usr/local/go/src/sync/cond.go:70 +0x85
golang.org/x/net/http2.(*pipe).Read(0xc000b4ac30, {0xc0011a0601, 0x5ff, 0x5ff})
        /go/pkg/mod/golang.org/x/net@v0.26.0/http2/pipe.go:76 +0xdf
golang.org/x/net/http2.transportResponseBody.Read({0x0?}, {0xc0011a0601?, 0xc000851ce0?, 0x411bbb?})
        /go/pkg/mod/golang.org/x/net@v0.26.0/http2/transport.go:2641 +0x65
encoding/json.(*Decoder).refill(0xc001302500)
        /usr/local/go/src/encoding/json/stream.go:165 +0x188
encoding/json.(*Decoder).readValue(0xc001302500)
        /usr/local/go/src/encoding/json/stream.go:140 +0x85
encoding/json.(*Decoder).Decode(0xc001302500, {0x1f414a0, 0xc000a9efd8})
        /usr/local/go/src/encoding/json/stream.go:63 +0x75
k8s.io/apimachinery/pkg/util/framer.(*jsonFrameReader).Read(0xc000c0d770, {0xc000bf8400, 0x400, 0x400})
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/util/framer/framer.go:152 +0x19c
k8s.io/apimachinery/pkg/runtime/serializer/streaming.(*decoder).Decode(0xc000bfc730, 0x0, {0x29bfff0, 0xc001165b00})
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/runtime/serializer/streaming/streaming.go:77 +0xa3
k8s.io/client-go/rest/watch.(*Decoder).Decode(0xc000fa63e0)
        /go/pkg/mod/k8s.io/client-go@v0.30.3/rest/watch/decoder.go:49 +0x4b
k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive(0xc000bb0e40)
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/watch/streamwatcher.go:105 +0xdb
created by k8s.io/apimachinery/pkg/watch.NewStreamWatcher in goroutine 188
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/watch/streamwatcher.go:76 +0x105

goroutine 6955598 [select, 5 minutes]:
golang.org/x/net/http2.(*clientStream).writeRequest(0xc000b4ac00, 0xc000bfa120, 0x0)
        /go/pkg/mod/golang.org/x/net@v0.26.0/http2/transport.go:1536 +0xa85
golang.org/x/net/http2.(*clientStream).doRequest(0xc000b4ac00, 0x921ce5?, 0xc000bb0c00?)
        /go/pkg/mod/golang.org/x/net@v0.26.0/http2/transport.go:1414 +0x56
created by golang.org/x/net/http2.(*ClientConn).roundTrip in goroutine 188
        /go/pkg/mod/golang.org/x/net@v0.26.0/http2/transport.go:1319 +0x3e5

goroutine 6952073 [sync.Cond.Wait]:
sync.runtime_notifyListWait(0xc000a6e348, 0x6)
        /usr/local/go/src/runtime/sema.go:569 +0x159
sync.(*Cond).Wait(0xc000bb1a40?)
        /usr/local/go/src/sync/cond.go:70 +0x85
golang.org/x/net/http2.(*pipe).Read(0xc000a6e330, {0xc000f94001, 0x5ff, 0x5ff})
        /go/pkg/mod/golang.org/x/net@v0.26.0/http2/pipe.go:76 +0xdf
golang.org/x/net/http2.transportResponseBody.Read({0x0?}, {0xc000f94001?, 0xc000853ce0?, 0x411bbb?})
        /go/pkg/mod/golang.org/x/net@v0.26.0/http2/transport.go:2641 +0x65
encoding/json.(*Decoder).refill(0xc000727b80)
        /usr/local/go/src/encoding/json/stream.go:165 +0x188
encoding/json.(*Decoder).readValue(0xc000727b80)
        /usr/local/go/src/encoding/json/stream.go:140 +0x85
encoding/json.(*Decoder).Decode(0xc000727b80, {0x1f414a0, 0xc000ba8cf0})
        /usr/local/go/src/encoding/json/stream.go:63 +0x75
k8s.io/apimachinery/pkg/util/framer.(*jsonFrameReader).Read(0xc000120960, {0xc0018f6000, 0x400, 0x400})
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/util/framer/framer.go:152 +0x19c
k8s.io/apimachinery/pkg/runtime/serializer/streaming.(*decoder).Decode(0xc000722960, 0x0, {0x29bfff0, 0xc00167ca40})
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/runtime/serializer/streaming/streaming.go:77 +0xa3
k8s.io/client-go/rest/watch.(*Decoder).Decode(0xc000fa77a0)
        /go/pkg/mod/k8s.io/client-go@v0.30.3/rest/watch/decoder.go:49 +0x4b
k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive(0xc000bb1ac0)
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/watch/streamwatcher.go:105 +0xdb
created by k8s.io/apimachinery/pkg/watch.NewStreamWatcher in goroutine 111
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/watch/streamwatcher.go:76 +0x105

goroutine 6971388 [sync.Cond.Wait]:
sync.runtime_notifyListWait(0xc001081848, 0x1)
        /usr/local/go/src/runtime/sema.go:569 +0x159
sync.(*Cond).Wait(0x0?)
        /usr/local/go/src/sync/cond.go:70 +0x85
golang.org/x/net/http2.(*pipe).Read(0xc001081830, {0xc0011a0c01, 0x5ff, 0x5ff})
        /go/pkg/mod/golang.org/x/net@v0.26.0/http2/pipe.go:76 +0xdf
golang.org/x/net/http2.transportResponseBody.Read({0x7f957fe5b798?}, {0xc0011a0c01?, 0xc00084fce0?, 0x411bbb?})
        /go/pkg/mod/golang.org/x/net@v0.26.0/http2/transport.go:2641 +0x65
encoding/json.(*Decoder).refill(0xc000e7fb80)
        /usr/local/go/src/encoding/json/stream.go:165 +0x188
encoding/json.(*Decoder).readValue(0xc000e7fb80)
        /usr/local/go/src/encoding/json/stream.go:140 +0x85
encoding/json.(*Decoder).Decode(0xc000e7fb80, {0x1f414a0, 0xc0002a5e00})
        /usr/local/go/src/encoding/json/stream.go:63 +0x75
k8s.io/apimachinery/pkg/util/framer.(*jsonFrameReader).Read(0xc000e49290, {0xc00109dc00, 0x400, 0x400})
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/util/framer/framer.go:152 +0x19c
k8s.io/apimachinery/pkg/runtime/serializer/streaming.(*decoder).Decode(0xc0007efc70, 0x0, {0x29bfff0, 0xc001458a40})
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/runtime/serializer/streaming/streaming.go:77 +0xa3
k8s.io/client-go/rest/watch.(*Decoder).Decode(0xc000885060)
        /go/pkg/mod/k8s.io/client-go@v0.30.3/rest/watch/decoder.go:49 +0x4b
k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive(0xc0003eda80)
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/watch/streamwatcher.go:105 +0xdb
created by k8s.io/apimachinery/pkg/watch.NewStreamWatcher in goroutine 195
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/watch/streamwatcher.go:76 +0x105

goroutine 6971387 [select, 1 minutes]:
golang.org/x/net/http2.(*clientStream).writeRequest(0xc001081800, 0xc001134ea0, 0x0)
        /go/pkg/mod/golang.org/x/net@v0.26.0/http2/transport.go:1536 +0xa85
golang.org/x/net/http2.(*clientStream).doRequest(0xc001081800, 0xc000784460?, 0xc00012d930?)
        /go/pkg/mod/golang.org/x/net@v0.26.0/http2/transport.go:1414 +0x56
created by golang.org/x/net/http2.(*ClientConn).roundTrip in goroutine 195
        /go/pkg/mod/golang.org/x/net@v0.26.0/http2/transport.go:1319 +0x3e5

goroutine 6972062 [select, 1 minutes]:
golang.org/x/net/http2.(*clientStream).writeRequest(0xc00133f800, 0xc00088a900, 0x0)
        /go/pkg/mod/golang.org/x/net@v0.26.0/http2/transport.go:1536 +0xa85
golang.org/x/net/http2.(*clientStream).doRequest(0xc00133f800, 0x921ce5?, 0xc00086bac0?)
        /go/pkg/mod/golang.org/x/net@v0.26.0/http2/transport.go:1414 +0x56
created by golang.org/x/net/http2.(*ClientConn).roundTrip in goroutine 2015999
        /go/pkg/mod/golang.org/x/net@v0.26.0/http2/transport.go:1319 +0x3e5

goroutine 2015997 [chan receive, 1327 minutes]:
k8s.io/client-go/tools/cache.(*sharedProcessor).run(0xc000168410, 0xc000bc83c0)
        /go/pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/shared_informer.go:807 +0x4d
k8s.io/client-go/tools/cache.(*sharedIndexInformer).Run.(*Group).StartWithChannel.func4()
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/util/wait/wait.go:55 +0x1b
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/util/wait/wait.go:72 +0x52
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start in goroutine 2015993
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/util/wait/wait.go:70 +0x73

goroutine 2015999 [select]:
k8s.io/client-go/tools/cache.watchHandler({0x0?, 0x0?, 0x3b5cf60?}, {0x29c0bd0, 0xc000de5c80}, {0x7f955fd08b28, 0xc000b7e370}, {0x29f1138, 0x21f1280}, 0x0, ...)
        /go/pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/reflector.go:732 +0x187
k8s.io/client-go/tools/cache.(*Reflector).watch(0xc0015b68c0, {0x0?, 0x0?}, 0xc000bc82a0, 0xc0014c7080)
        /go/pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/reflector.go:445 +0x545
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch(0xc0015b68c0, 0xc000bc82a0)
        /go/pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/reflector.go:365 +0x4d7
k8s.io/client-go/tools/cache.(*Reflector).Run.func1()
        /go/pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/reflector.go:298 +0x25
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x10?)
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/util/wait/backoff.go:226 +0x33
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc00084df50, {0x29bb420, 0xc000168640}, 0x1, 0xc000bc82a0)
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/util/wait/backoff.go:227 +0xaf
k8s.io/client-go/tools/cache.(*Reflector).Run(0xc0015b68c0, 0xc000bc82a0)
        /go/pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/reflector.go:297 +0x1c5
k8s.io/client-go/tools/cache.(*controller).Run.(*Group).StartWithChannel.func2()
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/util/wait/wait.go:55 +0x1b
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/util/wait/wait.go:72 +0x52
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start in goroutine 2015993
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/util/wait/wait.go:70 +0x73

goroutine 6955597 [sync.Cond.Wait]:
sync.runtime_notifyListWait(0xc000e8d548, 0x5)
        /usr/local/go/src/runtime/sema.go:569 +0x159
sync.(*Cond).Wait(0x2?)
        /usr/local/go/src/sync/cond.go:70 +0x85
golang.org/x/net/http2.(*pipe).Read(0xc000e8d530, {0xc001791498, 0x4, 0x4})
        /go/pkg/mod/golang.org/x/net@v0.26.0/http2/pipe.go:76 +0xdf
golang.org/x/net/http2.transportResponseBody.Read({0x0?}, {0xc001791498?, 0xc000fcbce0?, 0x411f65?})
        /go/pkg/mod/golang.org/x/net@v0.26.0/http2/transport.go:2641 +0x65
io.ReadAtLeast({0x7f95840c5818, 0xc000e8d500}, {0xc001791498, 0x4, 0x4}, 0x4)
        /usr/local/go/src/io/io.go:335 +0x90
k8s.io/apimachinery/pkg/util/framer.(*lengthDelimitedFrameReader).Read(0xc000ba8150, {0xc000bf8000, 0x400, 0x400})
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/util/framer/framer.go:76 +0x88
k8s.io/apimachinery/pkg/runtime/serializer/streaming.(*decoder).Decode(0xc000bfc690, 0x0, {0x29bfff0, 0xc0011650c0})
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/runtime/serializer/streaming/streaming.go:77 +0xa3
k8s.io/client-go/rest/watch.(*Decoder).Decode(0xc000fa61e0)
        /go/pkg/mod/k8s.io/client-go@v0.30.3/rest/watch/decoder.go:49 +0x4b
k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive(0xc000bb0c00)
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/watch/streamwatcher.go:105 +0xdb
created by k8s.io/apimachinery/pkg/watch.NewStreamWatcher in goroutine 2016113
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/watch/streamwatcher.go:76 +0x105

goroutine 6955596 [select, 5 minutes]:
golang.org/x/net/http2.(*clientStream).writeRequest(0xc000e8d500, 0xc0010f5b00, 0x0)
        /go/pkg/mod/golang.org/x/net@v0.26.0/http2/transport.go:1536 +0xa85
golang.org/x/net/http2.(*clientStream).doRequest(0xc000e8d500, 0xc000529540?, 0xc00071e260?)
        /go/pkg/mod/golang.org/x/net@v0.26.0/http2/transport.go:1414 +0x56
created by golang.org/x/net/http2.(*ClientConn).roundTrip in goroutine 2016113
        /go/pkg/mod/golang.org/x/net@v0.26.0/http2/transport.go:1319 +0x3e5

goroutine 2016113 [select]:
k8s.io/client-go/tools/cache.watchHandler({0x0?, 0x0?, 0x3b5cf60?}, {0x29c0bd0, 0xc000bb0c00}, {0x7f955fd08b28, 0xc000c26210}, {0x29f1138, 0x21f7f40}, 0x0, ...)
        /go/pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/reflector.go:732 +0x187
k8s.io/client-go/tools/cache.(*Reflector).watch(0xc00127b960, {0x0?, 0x0?}, 0xc000c6a600, 0xc0014c7920)
        /go/pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/reflector.go:445 +0x545
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch(0xc00127b960, 0xc000c6a600)
        /go/pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/reflector.go:365 +0x4d7
k8s.io/client-go/tools/cache.(*Reflector).Run.func1()
        /go/pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/reflector.go:298 +0x25
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x10?)
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/util/wait/backoff.go:226 +0x33
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000875f50, {0x29bb420, 0xc000118d70}, 0x1, 0xc000c6a600)
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/util/wait/backoff.go:227 +0xaf
k8s.io/client-go/tools/cache.(*Reflector).Run(0xc00127b960, 0xc000c6a600)
        /go/pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/reflector.go:297 +0x1c5
k8s.io/client-go/tools/cache.(*controller).Run.(*Group).StartWithChannel.func2()
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/util/wait/wait.go:55 +0x1b
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/util/wait/wait.go:72 +0x52
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start in goroutine 2016071
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/util/wait/wait.go:70 +0x73

goroutine 2016048 [chan receive, 1327 minutes]:
k8s.io/client-go/tools/cache.(*controller).Run.func1()
        /go/pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/controller.go:132 +0x25
created by k8s.io/client-go/tools/cache.(*controller).Run in goroutine 2016071
        /go/pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/controller.go:131 +0xa9

goroutine 2016047 [chan receive, 1327 minutes]:
k8s.io/client-go/tools/cache.(*sharedProcessor).run(0xc0000b2e10, 0xc000c6a6c0)
        /go/pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/shared_informer.go:807 +0x4d
k8s.io/client-go/tools/cache.(*sharedIndexInformer).Run.(*Group).StartWithChannel.func4()
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/util/wait/wait.go:55 +0x1b
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/util/wait/wait.go:72 +0x52
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start in goroutine 2016071
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/util/wait/wait.go:70 +0x73

goroutine 6972063 [sync.Cond.Wait]:
sync.runtime_notifyListWait(0xc00133f848, 0x1)
        /usr/local/go/src/runtime/sema.go:569 +0x159
sync.(*Cond).Wait(0x2?)
        /usr/local/go/src/sync/cond.go:70 +0x85
golang.org/x/net/http2.(*pipe).Read(0xc00133f830, {0xc000e020d8, 0x4, 0x4})
        /go/pkg/mod/golang.org/x/net@v0.26.0/http2/pipe.go:76 +0xdf
golang.org/x/net/http2.transportResponseBody.Read({0x0?}, {0xc000e020d8?, 0xc0010edce0?, 0x411f65?})
        /go/pkg/mod/golang.org/x/net@v0.26.0/http2/transport.go:2641 +0x65
io.ReadAtLeast({0x7f95840c5818, 0xc00133f800}, {0xc000e020d8, 0x4, 0x4}, 0x4)
        /usr/local/go/src/io/io.go:335 +0x90
k8s.io/apimachinery/pkg/util/framer.(*lengthDelimitedFrameReader).Read(0xc000880918, {0xc0018f7800, 0x400, 0x400})
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/util/framer/framer.go:76 +0x88
k8s.io/apimachinery/pkg/runtime/serializer/streaming.(*decoder).Decode(0xc000bfcb90, 0x0, {0x29bfff0, 0xc000d0e240})
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/runtime/serializer/streaming/streaming.go:77 +0xa3
k8s.io/client-go/rest/watch.(*Decoder).Decode(0xc001652f80)
        /go/pkg/mod/k8s.io/client-go@v0.30.3/rest/watch/decoder.go:49 +0x4b
k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive(0xc000de5c80)
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/watch/streamwatcher.go:105 +0xdb
created by k8s.io/apimachinery/pkg/watch.NewStreamWatcher in goroutine 2015999
        /go/pkg/mod/k8s.io/apimachinery@v0.30.3/pkg/watch/streamwatcher.go:76 +0x105

goroutine 6952072 [select, 6 minutes]:
golang.org/x/net/http2.(*clientStream).writeRequest(0xc000a6e300, 0xc000c4a480, 0x0)
        /go/pkg/mod/golang.org/x/net@v0.26.0/http2/transport.go:1536 +0xa85
golang.org/x/net/http2.(*clientStream).doRequest(0xc000a6e300, 0x5fa924?, 0xc000eaea20?)
        /go/pkg/mod/golang.org/x/net@v0.26.0/http2/transport.go:1414 +0x56
created by golang.org/x/net/http2.(*ClientConn).roundTrip in goroutine 111
        /go/pkg/mod/golang.org/x/net@v0.26.0/http2/transport.go:1319 +0x3e5

goroutine 2016101 [select, 104 minutes]:
k8s.io/client-go/tools/cache.(*Reflector).startResync(0xc00127b960, 0xc000c6a600, 0xc000bc8f60, 0xc0014c7920)
        /go/pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/reflector.go:377 +0x10f
created by k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch in goroutine 2016113
        /go/pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/reflector.go:364 +0x4ad
<end of log>
4

Nevermind this problem, I couldn’t find it with a search but I found this post Older backups are not getting deleted, with the keep argument specified in the related posts of this thread

Seems to be fixed in 1.18 Issue navigator - Percona JIRA

Therefore, closing this as solved :slight_smile:

5

glad to hear you sorted it out