percona operator ignores backup scheduler changes

I've enabled the scheduled backup for 5 Percona mongo clusters to run hourly with 5 minutes shift starting "00 * * * *" ending "20 * * * *".
Let it run 2 times to be sure it works. After successful backups on all clusters, I updated the CRD to run tasks at midnight with the same 5-minute shift.
PS: I've also removed the s3 secretkeys from k8s secret to see if aws role attached to a pod works.

this is perconaservermongodbbackups instances of an object looks like the next day. There were 9 backup runs at all before it starts to fail with no credentials found:
see the second post
note: the names are changed.

as you can see the schedule ignores the change I made. Here are the PerconaServerMongoDB instances:

user@unallocated:deploy$ kubectl  get PerconaServerMongoDB project-dev-cluster2-pmongo -o yaml
apiVersion: psmdb.percona.com/v1
kind: PerconaServerMongoDB
metadata:
  creationTimestamp: "2020-03-04T11:26:52Z"
...
  backup:
    coordinator:
      annotations:
        iam.amazonaws.com/role: arn:aws:iam::XXXXXXXXXXX:role/project-eks-test-project-dev-cluster2
...
    storages:
      s3-backup-bucket:
        s3:
          bucket: project-dev-cluster2-pmongo-dev-backup
          region: us-east-1
        type: s3
    tasks:
    - compressionType: gzip
      enabled: true
      name: daily
      schedule: 15 00 * * *
      storageName: s3-backup-bucket
....
status:
  conditions:
...
  - lastTransitionTime: "2020-03-04T11:28:18Z"
    status: "True"
    type: ReplsetReady
  - lastTransitionTime: "2020-03-04T18:12:39Z"
    message: 'reconcile backup storage config: get storage config: no credentials
      specified for the secret name s3-backup-bucket'
    reason: ErrorReconcile
    status: "True"
    type: Error
  message: 'Error: reconcile backup storage config: get storage config: no credentials
    specified for the secret name s3-backup-bucket'
  replsets:
    rs:
      initialized: true
      ready: 3
      size: 3
      status: ready
  state: Error



The second issue is with credentials - it looks like to be cached on the coordinator pod for a security token valid time (AWS default is 12 hours).

I'm not so worried about the second issue as for the first one.

Any ideas why the operator keeps creating k8s jobs on the old schedule?

Comments

  • OleksiiOleksii Contributor Active Member Participant
    user@unallocated:deploy$ kubectl  get perconaservermongodbbackups
    NAME                                         CLUSTER                         STORAGE            DESTINATION                 STATUS   COMPLETED   AGE
    cron-project-dev-clu-20200304151501-tau46   project-dev-cluster2-pmongo    s3-backup-bucket   2020-03-04T15:15:01Z.json   ready    18h         18h
    cron-project-dev-clu-20200304160004-iji5s   project-dev-cluster1-pmongo      s3-backup-bucket   2020-03-04T16:00:04Z.json   ready    18h         18h
    cron-project-dev-clu-20200304161508-bhaj3   project-dev-cluster2-pmongo    s3-backup-bucket   2020-03-04T16:15:08Z.json   ready    17h         17h
    cron-project-dev-clu-20200304170011-w95go   project-dev-cluster1-pmongo      s3-backup-bucket   2020-03-04T17:00:11Z.json   ready    17h         17h
    cron-project-dev-clu-20200304171505-3swh9   project-dev-cluster2-pmongo    s3-backup-bucket   2020-03-04T17:15:05Z.json   ready    16h         16h
    cron-project-dev-clu-20200304180007-ju3m7   project-dev-cluster1-pmongo      s3-backup-bucket   2020-03-04T18:00:08Z.json   ready    16h         16h
    cron-project-dev-clu-20200304181502-4hhud   project-dev-cluster2-pmongo    s3-backup-bucket   2020-03-04T18:15:02Z.json   ready    15h         15h
    cron-project-dev-clu-20200304190005-ju3m7   project-dev-cluster1-pmongo      s3-backup-bucket   2020-03-04T19:00:05Z.json   ready    15h         15h
    cron-project-dev-clu-20200304191508-4hhud   project-dev-cluster2-pmongo    s3-backup-bucket   2020-03-04T19:15:09Z.json   ready    14h         14h
    cron-project-dev-clu-20200304200001-ju3m7   project-dev-cluster1-pmongo      s3-backup-bucket   2020-03-04T20:00:02Z.json   ready    14h         14h
    cron-project-dev-clu-20200304201505-4hhud   project-dev-cluster2-pmongo    s3-backup-bucket   2020-03-04T20:15:06Z.json   ready    13h         13h
    cron-project-dev-clu-20200304210009-ju3m7   project-dev-cluster1-pmongo      s3-backup-bucket   2020-03-04T21:00:09Z.json   ready    13h         13h
    cron-project-dev-clu-20200304211502-4hhud   project-dev-cluster2-pmongo    s3-backup-bucket   2020-03-04T21:15:03Z.json   ready    12h         12h
    cron-project-dev-clu-20200304220005-ju3m7   project-dev-cluster1-pmongo      s3-backup-bucket   2020-03-04T22:00:05Z.json   ready    12h         12h
    cron-project-dev-clu-20200304221510-4hhud   project-dev-cluster2-pmongo    s3-backup-bucket   2020-03-04T22:15:10Z.json   ready    11h         11h
    cron-project-dev-clu-20200304230002-ju3m7   project-dev-cluster1-pmongo      s3-backup-bucket   2020-03-04T23:00:03Z.json   ready    11h         11h
    cron-project-dev-clu-20200304231507-4hhud   project-dev-cluster2-pmongo    s3-backup-bucket   2020-03-04T23:15:07Z.json   ready    10h         10h
    cron-project-dev-clu-20200305000009-ju3m7   project-dev-cluster1-pmongo      s3-backup-bucket                                                    10h
    cron-project-dev-clu-20200305001504-4hhud   project-dev-cluster2-pmongo    s3-backup-bucket                                                    9h
    cron-project-dev-clu-20200305010007-ju3m7   project-dev-cluster1-pmongo      s3-backup-bucket                                                    9h
    cron-project-dev-clu-20200305011511-4hhud   project-dev-cluster2-pmongo    s3-backup-bucket                                                    8h
    cron-project-dev-clu-20200305020004-ju3m7   project-dev-cluster1-pmongo      s3-backup-bucket                                                    8h
    cron-project-dev-clu-20200305021508-4hhud   project-dev-cluster2-pmongo    s3-backup-bucket                                                    7h48m
    cron-project-dev-clu-20200305030011-ju3m7   project-dev-cluster1-pmongo      s3-backup-bucket                                                    7h3m
    cron-project-dev-clu-20200305031505-4hhud   project-dev-cluster2-pmongo    s3-backup-bucket                                                    6h48m
    cron-project-dev-clu-20200305040008-ju3m7   project-dev-cluster1-pmongo      s3-backup-bucket                                                    6h3m
    cron-project-dev-clu-20200305041502-4hhud   project-dev-cluster2-pmongo    s3-backup-bucket                                                    5h48m
    cron-project-dev-clu-20200305050005-ju3m7   project-dev-cluster1-pmongo      s3-backup-bucket                                                    5h3m
    cron-project-dev-clu-20200305051509-4hhud   project-dev-cluster2-pmongo    s3-backup-bucket                                                    4h48m
    cron-project-dev-clu-20200305060002-ju3m7   project-dev-cluster1-pmongo      s3-backup-bucket                                                    4h3m
    cron-project-dev-clu-20200305061506-4hhud   project-dev-cluster2-pmongo    s3-backup-bucket                                                    3h48m
    cron-project-dev-clu-20200305070010-ju3m7   project-dev-cluster1-pmongo      s3-backup-bucket                                                    3h3m
    cron-project-dev-clu-20200305071503-4hhud   project-dev-cluster2-pmongo    s3-backup-bucket                                                    168m
    cron-project-dev-clu-20200305080006-ju3m7   project-dev-cluster1-pmongo      s3-backup-bucket                                                    123m
    cron-project-dev-clu-20200305081510-4hhud   project-dev-cluster2-pmongo    s3-backup-bucket                                                    108m
    cron-project-dev-clu-20200305090004-ju3m7   project-dev-cluster1-pmongo      s3-backup-bucket                                                    63m
    cron-project-dev-clu-20200305091508-4hhud   project-dev-cluster2-pmongo    s3-backup-bucket                                                    48m
    cron-project-dev-clu-20200305100010-ju3m7   project-dev-cluster1-pmongo      s3-backup-bucket                                                    3m17s
    cron-project-dev-clu-20200304152003-b78zu   project-dev-cluster3-pmongo       s3-backup-bucket   2020-03-04T15:20:03Z.json   ready    18h         18h
    cron-project-dev-clu-20200304162009-5l8id   project-dev-cluster3-pmongo       s3-backup-bucket   2020-03-04T16:20:09Z.json   ready    17h         17h
    cron-project-dev-clu-20200304172006-wpcv7   project-dev-cluster3-pmongo       s3-backup-bucket   2020-03-04T17:20:06Z.json   ready    16h         16h
    cron-project-dev-clu-20200304182003-irvad   project-dev-cluster3-pmongo       s3-backup-bucket   2020-03-04T18:20:03Z.json   ready    15h         15h
    cron-project-dev-clu-20200304192010-irvad   project-dev-cluster3-pmongo       s3-backup-bucket   2020-03-04T19:20:10Z.json   ready    14h         14h
    cron-project-dev-clu-20200304202007-irvad   project-dev-cluster3-pmongo       s3-backup-bucket   2020-03-04T20:20:07Z.json   ready    13h         13h
    cron-project-dev-clu-20200304212004-irvad   project-dev-cluster3-pmongo       s3-backup-bucket   2020-03-04T21:20:04Z.json   ready    12h         12h
    cron-project-dev-clu-20200304222001-irvad   project-dev-cluster3-pmongo       s3-backup-bucket   2020-03-04T22:20:01Z.json   ready    11h         11h
    cron-project-dev-clu-20200304232008-irvad   project-dev-cluster3-pmongo       s3-backup-bucket   2020-03-04T23:20:08Z.json   ready    10h         10h
    cron-project-dev-clu-20200305002005-irvad   project-dev-cluster3-pmongo       s3-backup-bucket                                                    9h
    cron-project-dev-clu-20200305012002-irvad   project-dev-cluster3-pmongo       s3-backup-bucket                                                    8h
    cron-project-dev-clu-20200305022009-irvad   project-dev-cluster3-pmongo       s3-backup-bucket                                                    7h43m
    cron-project-dev-clu-20200305032007-irvad   project-dev-cluster3-pmongo       s3-backup-bucket                                                    6h43m
    cron-project-dev-clu-20200305042003-irvad   project-dev-cluster3-pmongo       s3-backup-bucket                                                    5h43m
    cron-project-dev-clu-20200305052011-irvad   project-dev-cluster3-pmongo       s3-backup-bucket                                                    4h43m
    cron-project-dev-clu-20200305062008-irvad   project-dev-cluster3-pmongo       s3-backup-bucket                                                    3h43m
    cron-project-dev-clu-20200305072005-irvad   project-dev-cluster3-pmongo       s3-backup-bucket                                                    163m
    cron-project-dev-clu-20200305082002-irvad   project-dev-cluster3-pmongo       s3-backup-bucket                                                    103m
    cron-project-dev-clu-20200305092009-irvad   project-dev-cluster3-pmongo       s3-backup-bucket                                                    43m
    


    and more ...
  • OleksiiOleksii Contributor Active Member Participant
    cron-project-dev-clu-20200304161007-j0hq0   project-dev-cluster4-pmongo      s3-backup-bucket   2020-03-04T16:10:07Z.json   ready    17h         17h
    cron-project-dev-clu-20200304171003-uemxi   project-dev-cluster4-pmongo      s3-backup-bucket   2020-03-04T17:10:03Z.json   ready    16h         16h
    cron-project-dev-clu-20200304181010-kzgn9   project-dev-cluster4-pmongo      s3-backup-bucket   2020-03-04T18:10:11Z.json   ready    15h         15h
    cron-project-dev-clu-20200304191007-laoir   project-dev-cluster4-pmongo      s3-backup-bucket   2020-03-04T19:10:07Z.json   ready    14h         14h
    cron-project-dev-clu-20200304201004-laoir   project-dev-cluster4-pmongo      s3-backup-bucket   2020-03-04T20:10:04Z.json   ready    13h         13h
    cron-project-dev-clu-20200304211001-laoir   project-dev-cluster4-pmongo      s3-backup-bucket   2020-03-04T21:10:02Z.json   ready    12h         12h
    cron-project-dev-clu-20200304221008-laoir   project-dev-cluster4-pmongo      s3-backup-bucket   2020-03-04T22:10:08Z.json   ready    11h         11h
    cron-project-dev-clu-20200304231005-laoir   project-dev-cluster4-pmongo      s3-backup-bucket   2020-03-04T23:10:05Z.json   ready    10h         10h
    cron-project-dev-clu-20200305001002-laoir   project-dev-cluster4-pmongo      s3-backup-bucket                                                    9h
    cron-project-dev-clu-20200305011009-laoir   project-dev-cluster4-pmongo      s3-backup-bucket                                                    8h
    cron-project-dev-clu-20200305021006-laoir   project-dev-cluster4-pmongo      s3-backup-bucket                                                    7h53m
    cron-project-dev-clu-20200305031004-laoir   project-dev-cluster4-pmongo      s3-backup-bucket                                                    6h53m
    cron-project-dev-clu-20200305041011-laoir   project-dev-cluster4-pmongo      s3-backup-bucket                                                    5h53m
    cron-project-dev-clu-20200305051008-laoir   project-dev-cluster4-pmongo      s3-backup-bucket                                                    4h53m
    cron-project-dev-clu-20200305061005-laoir   project-dev-cluster4-pmongo      s3-backup-bucket                                                    3h53m
    cron-project-dev-clu-20200305071002-laoir   project-dev-cluster4-pmongo      s3-backup-bucket                                                    173m
    cron-project-dev-clu-20200305081009-laoir   project-dev-cluster4-pmongo      s3-backup-bucket                                                    113m
    cron-project-dev-clu-20200305091006-laoir   project-dev-cluster4-pmongo      s3-backup-bucket                                                    53m
    cron-project-dev-clu-20200304160505-6ach4   project-dev-cluster5-pmongo   s3-backup-bucket   2020-03-04T16:05:05Z.json   ready    17h         17h
    cron-project-dev-clu-20200304170502-m16h2   project-dev-cluster5-pmongo   s3-backup-bucket   2020-03-04T17:05:02Z.json   ready    16h         16h
    cron-project-dev-clu-20200304180509-ztm5v   project-dev-cluster5-pmongo   s3-backup-bucket   2020-03-04T18:05:09Z.json   ready    15h         15h
    cron-project-dev-clu-20200304190506-bp5fz   project-dev-cluster5-pmongo   s3-backup-bucket   2020-03-04T19:05:06Z.json   ready    14h         14h
    cron-project-dev-clu-20200304200503-bp5fz   project-dev-cluster5-pmongo   s3-backup-bucket   2020-03-04T20:05:03Z.json   ready    13h         13h
    cron-project-dev-clu-20200304210510-bp5fz   project-dev-cluster5-pmongo   s3-backup-bucket   2020-03-04T21:05:10Z.json   ready    12h         12h
    cron-project-dev-clu-20200304220507-bp5fz   project-dev-cluster5-pmongo   s3-backup-bucket   2020-03-04T22:05:07Z.json   ready    11h         11h
    cron-project-dev-clu-20200304230504-bp5fz   project-dev-cluster5-pmongo   s3-backup-bucket   2020-03-04T23:05:04Z.json   ready    10h         10h
    cron-project-dev-clu-20200305000511-bp5fz   project-dev-cluster5-pmongo   s3-backup-bucket                                                    9h
    cron-project-dev-clu-20200305010508-bp5fz   project-dev-cluster5-pmongo   s3-backup-bucket                                                    8h
    cron-project-dev-clu-20200305020505-bp5fz   project-dev-cluster5-pmongo   s3-backup-bucket                                                    7h58m
    cron-project-dev-clu-20200305030502-bp5fz   project-dev-cluster5-pmongo   s3-backup-bucket                                                    6h58m
    cron-project-dev-clu-20200305040509-bp5fz   project-dev-cluster5-pmongo   s3-backup-bucket                                                    5h58m
    cron-project-dev-clu-20200305050506-bp5fz   project-dev-cluster5-pmongo   s3-backup-bucket                                                    4h58m
    cron-project-dev-clu-20200305060504-bp5fz   project-dev-cluster5-pmongo   s3-backup-bucket                                                    3h58m
    cron-project-dev-clu-20200305070510-bp5fz   project-dev-cluster5-pmongo   s3-backup-bucket                                                    178m
    cron-project-dev-clu-20200305080508-bp5fz   project-dev-cluster5-pmongo   s3-backup-bucket                                                    118m
    cron-project-dev-clu-20200305090505-bp5fz   project-dev-cluster5-pmongo   s3-backup-bucket                                                    58m
    user@unallocated:deploy$
    
  • OleksiiOleksii Contributor Active Member Participant
    some extra info.
    user@unallocated:~$ kubectl  get PerconaServerMongoDB
    NAME                            STATUS   AGE
    project-dev-cluster2-pmongo    error    24h
    project-dev-cluster1-pmongo      error    25h
    project-dev-cluster3-pmongo       error    24h
    project-dev-cluster5-pmongo      error    24h
    project-dev-cluster4-pmongo   error    24h
    user@unallocated:~$
    





    After i put the AWS keys back in k8s secret and updated the CRD with credential the status changed to ready and no new hourly perconaservermongodbbackups appear:
    user@unallocated:~$ kubectl  get PerconaServerMongoDB
    NAME                            STATUS   AGE
    project-dev-cluster2-pmongo    ready    25h
    project-dev-cluster1-pmongo      ready    26h
    project-dev-cluster3-pmongo       ready    25h
    project-dev-cluster5-pmongo      ready    25h
    project-dev-cluster4-pmongo   ready    25h
    user@unallocated:~$
    


    at midnight the new one should appear... will keep this thread updated.
  • lorraine.pocklingtonlorraine.pocklington Percona Community Manager Reader Mentor
    Hi again Oleksii, appreciated.
    The team have asked if you would be prepared to create a bug in Jira for us for this issue? You'll be able to follow along on the progress then (alongside the other one you made to day).
    Here's the link again: https://jira.percona.com/secure/RapidBoard.jspa?rapidView=127&projectKey=K8SPSMDB
Sign In or Register to comment.

MySQL, InnoDB, MariaDB and MongoDB are trademarks of their respective owners.
Copyright ©2005 - 2020 Percona LLC. All rights reserved.