I’ve enabled the scheduled backup for 5 Percona mongo clusters to run hourly with 5 minutes shift starting “00 * * * *” ending “20 * * * *”.
Let it run 2 times to be sure it works. After successful backups on all clusters, I updated the CRD to run tasks at midnight with the same 5-minute shift.
PS: I’ve also removed the s3 secretkeys from k8s secret to see if aws role attached to a pod works.
this is perconaservermongodbbackups instances of an object looks like the next day. There were 9 backup runs at all before it starts to fail with no credentials found:
see the second post
note: the names are changed.
as you can see the schedule ignores the change I made. Here are the PerconaServerMongoDB instances:
user@unallocated:deploy$ kubectl get PerconaServerMongoDB project-dev-cluster2-pmongo -o yaml
apiVersion: psmdb.percona.com/v1
kind: PerconaServerMongoDB
metadata:
creationTimestamp: "2020-03-04T11:26:52Z"
...
backup:
coordinator:
annotations:
iam.amazonaws.com/role: arn:aws:iam::XXXXXXXXXXX:role/project-eks-test-project-dev-cluster2
...
storages:
s3-backup-bucket:
s3:
bucket: project-dev-cluster2-pmongo-dev-backup
region: us-east-1
type: s3
tasks:
- compressionType: gzip
enabled: true
name: daily
schedule: 15 00 * * *
storageName: s3-backup-bucket
....
status:
conditions:
...
- lastTransitionTime: "2020-03-04T11:28:18Z"
status: "True"
type: ReplsetReady
- lastTransitionTime: "2020-03-04T18:12:39Z"
message: 'reconcile backup storage config: get storage config: no credentials
specified for the secret name s3-backup-bucket'
reason: ErrorReconcile
status: "True"
type: Error
message: 'Error: reconcile backup storage config: get storage config: no credentials
specified for the secret name s3-backup-bucket'
replsets:
rs:
initialized: true
ready: 3
size: 3
status: ready
state: Error
The second issue is with credentials - it looks like to be cached on the coordinator pod for a security token valid time (AWS default is 12 hours).
I’m not so worried about the second issue as for the first one.
Any ideas why the operator keeps creating k8s jobs on the old schedule?
user@unallocated:deploy$ kubectl get perconaservermongodbbackups
NAME CLUSTER STORAGE DESTINATION STATUS COMPLETED AGE
cron-project-dev-clu-20200304151501-tau46 project-dev-cluster2-pmongo s3-backup-bucket 2020-03-04T15:15:01Z.json ready 18h 18h
cron-project-dev-clu-20200304160004-iji5s project-dev-cluster1-pmongo s3-backup-bucket 2020-03-04T16:00:04Z.json ready 18h 18h
cron-project-dev-clu-20200304161508-bhaj3 project-dev-cluster2-pmongo s3-backup-bucket 2020-03-04T16:15:08Z.json ready 17h 17h
cron-project-dev-clu-20200304170011-w95go project-dev-cluster1-pmongo s3-backup-bucket 2020-03-04T17:00:11Z.json ready 17h 17h
cron-project-dev-clu-20200304171505-3swh9 project-dev-cluster2-pmongo s3-backup-bucket 2020-03-04T17:15:05Z.json ready 16h 16h
cron-project-dev-clu-20200304180007-ju3m7 project-dev-cluster1-pmongo s3-backup-bucket 2020-03-04T18:00:08Z.json ready 16h 16h
cron-project-dev-clu-20200304181502-4hhud project-dev-cluster2-pmongo s3-backup-bucket 2020-03-04T18:15:02Z.json ready 15h 15h
cron-project-dev-clu-20200304190005-ju3m7 project-dev-cluster1-pmongo s3-backup-bucket 2020-03-04T19:00:05Z.json ready 15h 15h
cron-project-dev-clu-20200304191508-4hhud project-dev-cluster2-pmongo s3-backup-bucket 2020-03-04T19:15:09Z.json ready 14h 14h
cron-project-dev-clu-20200304200001-ju3m7 project-dev-cluster1-pmongo s3-backup-bucket 2020-03-04T20:00:02Z.json ready 14h 14h
cron-project-dev-clu-20200304201505-4hhud project-dev-cluster2-pmongo s3-backup-bucket 2020-03-04T20:15:06Z.json ready 13h 13h
cron-project-dev-clu-20200304210009-ju3m7 project-dev-cluster1-pmongo s3-backup-bucket 2020-03-04T21:00:09Z.json ready 13h 13h
cron-project-dev-clu-20200304211502-4hhud project-dev-cluster2-pmongo s3-backup-bucket 2020-03-04T21:15:03Z.json ready 12h 12h
cron-project-dev-clu-20200304220005-ju3m7 project-dev-cluster1-pmongo s3-backup-bucket 2020-03-04T22:00:05Z.json ready 12h 12h
cron-project-dev-clu-20200304221510-4hhud project-dev-cluster2-pmongo s3-backup-bucket 2020-03-04T22:15:10Z.json ready 11h 11h
cron-project-dev-clu-20200304230002-ju3m7 project-dev-cluster1-pmongo s3-backup-bucket 2020-03-04T23:00:03Z.json ready 11h 11h
cron-project-dev-clu-20200304231507-4hhud project-dev-cluster2-pmongo s3-backup-bucket 2020-03-04T23:15:07Z.json ready 10h 10h
cron-project-dev-clu-20200305000009-ju3m7 project-dev-cluster1-pmongo s3-backup-bucket 10h
cron-project-dev-clu-20200305001504-4hhud project-dev-cluster2-pmongo s3-backup-bucket 9h
cron-project-dev-clu-20200305010007-ju3m7 project-dev-cluster1-pmongo s3-backup-bucket 9h
cron-project-dev-clu-20200305011511-4hhud project-dev-cluster2-pmongo s3-backup-bucket 8h
cron-project-dev-clu-20200305020004-ju3m7 project-dev-cluster1-pmongo s3-backup-bucket 8h
cron-project-dev-clu-20200305021508-4hhud project-dev-cluster2-pmongo s3-backup-bucket 7h48m
cron-project-dev-clu-20200305030011-ju3m7 project-dev-cluster1-pmongo s3-backup-bucket 7h3m
cron-project-dev-clu-20200305031505-4hhud project-dev-cluster2-pmongo s3-backup-bucket 6h48m
cron-project-dev-clu-20200305040008-ju3m7 project-dev-cluster1-pmongo s3-backup-bucket 6h3m
cron-project-dev-clu-20200305041502-4hhud project-dev-cluster2-pmongo s3-backup-bucket 5h48m
cron-project-dev-clu-20200305050005-ju3m7 project-dev-cluster1-pmongo s3-backup-bucket 5h3m
cron-project-dev-clu-20200305051509-4hhud project-dev-cluster2-pmongo s3-backup-bucket 4h48m
cron-project-dev-clu-20200305060002-ju3m7 project-dev-cluster1-pmongo s3-backup-bucket 4h3m
cron-project-dev-clu-20200305061506-4hhud project-dev-cluster2-pmongo s3-backup-bucket 3h48m
cron-project-dev-clu-20200305070010-ju3m7 project-dev-cluster1-pmongo s3-backup-bucket 3h3m
cron-project-dev-clu-20200305071503-4hhud project-dev-cluster2-pmongo s3-backup-bucket 168m
cron-project-dev-clu-20200305080006-ju3m7 project-dev-cluster1-pmongo s3-backup-bucket 123m
cron-project-dev-clu-20200305081510-4hhud project-dev-cluster2-pmongo s3-backup-bucket 108m
cron-project-dev-clu-20200305090004-ju3m7 project-dev-cluster1-pmongo s3-backup-bucket 63m
cron-project-dev-clu-20200305091508-4hhud project-dev-cluster2-pmongo s3-backup-bucket 48m
cron-project-dev-clu-20200305100010-ju3m7 project-dev-cluster1-pmongo s3-backup-bucket 3m17s
cron-project-dev-clu-20200304152003-b78zu project-dev-cluster3-pmongo s3-backup-bucket 2020-03-04T15:20:03Z.json ready 18h 18h
cron-project-dev-clu-20200304162009-5l8id project-dev-cluster3-pmongo s3-backup-bucket 2020-03-04T16:20:09Z.json ready 17h 17h
cron-project-dev-clu-20200304172006-wpcv7 project-dev-cluster3-pmongo s3-backup-bucket 2020-03-04T17:20:06Z.json ready 16h 16h
cron-project-dev-clu-20200304182003-irvad project-dev-cluster3-pmongo s3-backup-bucket 2020-03-04T18:20:03Z.json ready 15h 15h
cron-project-dev-clu-20200304192010-irvad project-dev-cluster3-pmongo s3-backup-bucket 2020-03-04T19:20:10Z.json ready 14h 14h
cron-project-dev-clu-20200304202007-irvad project-dev-cluster3-pmongo s3-backup-bucket 2020-03-04T20:20:07Z.json ready 13h 13h
cron-project-dev-clu-20200304212004-irvad project-dev-cluster3-pmongo s3-backup-bucket 2020-03-04T21:20:04Z.json ready 12h 12h
cron-project-dev-clu-20200304222001-irvad project-dev-cluster3-pmongo s3-backup-bucket 2020-03-04T22:20:01Z.json ready 11h 11h
cron-project-dev-clu-20200304232008-irvad project-dev-cluster3-pmongo s3-backup-bucket 2020-03-04T23:20:08Z.json ready 10h 10h
cron-project-dev-clu-20200305002005-irvad project-dev-cluster3-pmongo s3-backup-bucket 9h
cron-project-dev-clu-20200305012002-irvad project-dev-cluster3-pmongo s3-backup-bucket 8h
cron-project-dev-clu-20200305022009-irvad project-dev-cluster3-pmongo s3-backup-bucket 7h43m
cron-project-dev-clu-20200305032007-irvad project-dev-cluster3-pmongo s3-backup-bucket 6h43m
cron-project-dev-clu-20200305042003-irvad project-dev-cluster3-pmongo s3-backup-bucket 5h43m
cron-project-dev-clu-20200305052011-irvad project-dev-cluster3-pmongo s3-backup-bucket 4h43m
cron-project-dev-clu-20200305062008-irvad project-dev-cluster3-pmongo s3-backup-bucket 3h43m
cron-project-dev-clu-20200305072005-irvad project-dev-cluster3-pmongo s3-backup-bucket 163m
cron-project-dev-clu-20200305082002-irvad project-dev-cluster3-pmongo s3-backup-bucket 103m
cron-project-dev-clu-20200305092009-irvad project-dev-cluster3-pmongo s3-backup-bucket 43m
and more …
cron-project-dev-clu-20200304161007-j0hq0 project-dev-cluster4-pmongo s3-backup-bucket 2020-03-04T16:10:07Z.json ready 17h 17h
cron-project-dev-clu-20200304171003-uemxi project-dev-cluster4-pmongo s3-backup-bucket 2020-03-04T17:10:03Z.json ready 16h 16h
cron-project-dev-clu-20200304181010-kzgn9 project-dev-cluster4-pmongo s3-backup-bucket 2020-03-04T18:10:11Z.json ready 15h 15h
cron-project-dev-clu-20200304191007-laoir project-dev-cluster4-pmongo s3-backup-bucket 2020-03-04T19:10:07Z.json ready 14h 14h
cron-project-dev-clu-20200304201004-laoir project-dev-cluster4-pmongo s3-backup-bucket 2020-03-04T20:10:04Z.json ready 13h 13h
cron-project-dev-clu-20200304211001-laoir project-dev-cluster4-pmongo s3-backup-bucket 2020-03-04T21:10:02Z.json ready 12h 12h
cron-project-dev-clu-20200304221008-laoir project-dev-cluster4-pmongo s3-backup-bucket 2020-03-04T22:10:08Z.json ready 11h 11h
cron-project-dev-clu-20200304231005-laoir project-dev-cluster4-pmongo s3-backup-bucket 2020-03-04T23:10:05Z.json ready 10h 10h
cron-project-dev-clu-20200305001002-laoir project-dev-cluster4-pmongo s3-backup-bucket 9h
cron-project-dev-clu-20200305011009-laoir project-dev-cluster4-pmongo s3-backup-bucket 8h
cron-project-dev-clu-20200305021006-laoir project-dev-cluster4-pmongo s3-backup-bucket 7h53m
cron-project-dev-clu-20200305031004-laoir project-dev-cluster4-pmongo s3-backup-bucket 6h53m
cron-project-dev-clu-20200305041011-laoir project-dev-cluster4-pmongo s3-backup-bucket 5h53m
cron-project-dev-clu-20200305051008-laoir project-dev-cluster4-pmongo s3-backup-bucket 4h53m
cron-project-dev-clu-20200305061005-laoir project-dev-cluster4-pmongo s3-backup-bucket 3h53m
cron-project-dev-clu-20200305071002-laoir project-dev-cluster4-pmongo s3-backup-bucket 173m
cron-project-dev-clu-20200305081009-laoir project-dev-cluster4-pmongo s3-backup-bucket 113m
cron-project-dev-clu-20200305091006-laoir project-dev-cluster4-pmongo s3-backup-bucket 53m
cron-project-dev-clu-20200304160505-6ach4 project-dev-cluster5-pmongo s3-backup-bucket 2020-03-04T16:05:05Z.json ready 17h 17h
cron-project-dev-clu-20200304170502-m16h2 project-dev-cluster5-pmongo s3-backup-bucket 2020-03-04T17:05:02Z.json ready 16h 16h
cron-project-dev-clu-20200304180509-ztm5v project-dev-cluster5-pmongo s3-backup-bucket 2020-03-04T18:05:09Z.json ready 15h 15h
cron-project-dev-clu-20200304190506-bp5fz project-dev-cluster5-pmongo s3-backup-bucket 2020-03-04T19:05:06Z.json ready 14h 14h
cron-project-dev-clu-20200304200503-bp5fz project-dev-cluster5-pmongo s3-backup-bucket 2020-03-04T20:05:03Z.json ready 13h 13h
cron-project-dev-clu-20200304210510-bp5fz project-dev-cluster5-pmongo s3-backup-bucket 2020-03-04T21:05:10Z.json ready 12h 12h
cron-project-dev-clu-20200304220507-bp5fz project-dev-cluster5-pmongo s3-backup-bucket 2020-03-04T22:05:07Z.json ready 11h 11h
cron-project-dev-clu-20200304230504-bp5fz project-dev-cluster5-pmongo s3-backup-bucket 2020-03-04T23:05:04Z.json ready 10h 10h
cron-project-dev-clu-20200305000511-bp5fz project-dev-cluster5-pmongo s3-backup-bucket 9h
cron-project-dev-clu-20200305010508-bp5fz project-dev-cluster5-pmongo s3-backup-bucket 8h
cron-project-dev-clu-20200305020505-bp5fz project-dev-cluster5-pmongo s3-backup-bucket 7h58m
cron-project-dev-clu-20200305030502-bp5fz project-dev-cluster5-pmongo s3-backup-bucket 6h58m
cron-project-dev-clu-20200305040509-bp5fz project-dev-cluster5-pmongo s3-backup-bucket 5h58m
cron-project-dev-clu-20200305050506-bp5fz project-dev-cluster5-pmongo s3-backup-bucket 4h58m
cron-project-dev-clu-20200305060504-bp5fz project-dev-cluster5-pmongo s3-backup-bucket 3h58m
cron-project-dev-clu-20200305070510-bp5fz project-dev-cluster5-pmongo s3-backup-bucket 178m
cron-project-dev-clu-20200305080508-bp5fz project-dev-cluster5-pmongo s3-backup-bucket 118m
cron-project-dev-clu-20200305090505-bp5fz project-dev-cluster5-pmongo s3-backup-bucket 58m
user@unallocated:deploy$
some extra info.
user@unallocated:~$ kubectl get PerconaServerMongoDB
NAME STATUS AGE
project-dev-cluster2-pmongo error 24h
project-dev-cluster1-pmongo error 25h
project-dev-cluster3-pmongo error 24h
project-dev-cluster5-pmongo error 24h
project-dev-cluster4-pmongo error 24h
user@unallocated:~$
After i put the AWS keys back in k8s secret and updated the CRD with credential the status changed to ready and no new hourly perconaservermongodbbackups appear:
user@unallocated:~$ kubectl get PerconaServerMongoDB
NAME STATUS AGE
project-dev-cluster2-pmongo ready 25h
project-dev-cluster1-pmongo ready 26h
project-dev-cluster3-pmongo ready 25h
project-dev-cluster5-pmongo ready 25h
project-dev-cluster4-pmongo ready 25h
user@unallocated:~$
at midnight the new one should appear… will keep this thread updated.
Hi again Oleksii, appreciated.
The team have asked if you would be prepared to create a bug in Jira for us for this issue? You’ll be able to follow along on the progress then (alongside the other one you made to day).
Here’s the link again: [URL]Log in - Percona JIRA