I deployed the same perfectly working PostgresCluster from the non istio namespace to an Istio enabled namespace.
Managed to solve all other connection issues but it seems that the S3 backup doesn’t work
I know this is not a percona postgres issue but I thought that someone might have hit the same wall as I did
Sorry for the late response, actually had an ugly motorcycle accident after my last post.
So, I can start and describe how postgres backup behaves without Istio injection.
This is my sample test backup code for postgres for the namespace without istio injection:
No problem here, backup job pods get started at the scheduled time, jobs are completed and waiting for the next scheduled job and the PergonaPGBackups match all the backups done and jobs started(Sounds obviously, but not for Istio injected postgres)
NAME CLUSTER REPO DESTINATION STATUS TYPE COMPLETED AGE
service-1-pg-db-backup-ct88-5mv2v service-1-pg-db repo1 Succeeded incremental 96m 99m
service-1-pg-db-repo1-full-br5c9 service-1-pg-db repo1 Succeeded full 30m 30m
service-1-pg-db-repo1-full-t88h6 service-1-pg-db repo1 Succeeded full 90m 90m
service-1-pg-db-repo2-full-c47kh service-1-pg-db repo2 s3://b948b94bbae997066c0b3a4a6b25be3477b482b6cf23267b94ad1b92 Succeeded full 14m 15m
service-1-pg-db-repo2-full-k5k5n service-1-pg-db repo2 s3://b948b94bbae997066c0b3a4a6b25be3477b482b6cf23267b94ad1b92 Succeeded full 67m 75m
Now, regarding the Istio injected namespace, beside the Istio service entries for and peer authentication all postgres containers and minio endpoint, I have this backup config:
This is a new test postgres cluster and the reason why there is only S3 backup at this point, is because if I deploy the cluster with both S3 and PV backup repos, S3 backup won’t work, pgback rest will say that repo1(S3) is not ok when executing pgbackrest info. But If I deploy S3 backup only at the beginning, and make an upgrade and add repo2 with PV backup, they will both “work”.
The issue afterwards, is that when one backup job starts, it will spawn a job pod with 2 containers, istio-proxy and pgbackrest. When the pgbackrest completes, the job won’t get completed and I will have a remaining backup job pod with only istio-proxy running and because of this, no other backup job will start, even if I can see many PerconaPGBackup resources started.
With S3 backup, same issue like with the PV backup, but in this case, the PerconaPGBackup jobs are stuck in running state and backups jobs are triggered randomly though they are supposed to run only at minute 30. I can see backups on the S3 bucket as well:
Considering that I only have S3 backup configured, there is no pgbackrest pod to use pgbackrest info and check backups status
Actually, I found out this is a known issue, for which Istio released a fix but it was not implemented in Kubernetes. The fix would bee to call /quitquitquit on envoy api to terminate istio-proxy container after the backup job is completed, on which I am looking for a proper solution at this moment:
More details here:
So the solution would be to have a custom made pgbackrest container that calls Pilot agent /quitquitquit endpoint to terminate istio-proxy
I tried to see how it behaves if I would add sidecar.istio.io/inject: “false” annotation to pgbackrest but nothing happens, no annotation is added to backup pod: