Description / Use Case:
In enterprise environments, it is standard practice to use Mutating Webhooks (like OPA Gatekeeper or Kyverno) to enforce mandatory organizational labels (e.g., billing codes, monitoring flags) on all resources across a namespace.
Currently, when the Percona Operator generates child resources like StatefulSets and PodDisruptionBudgets, our mutating webhooks successfully inject our mandatory labels into them. However, the Operator’s reconciliation loop detects these extra labels as “drift,” strips them off, and triggers an infinite update loop as the webhook instantly re-injects them.
Current Behavior:
Because the Operator strips unexpected labels from PDBs and STSs, the cluster gets trapped in a race condition.
-
Operator creates/updates PDB without our custom labels.
-
Gatekeeper intercepts and injects the labels.
-
Operator runs sync, detects drift on the PDB, and sends an update to remove the labels.
-
Gatekeeper intercepts the update and re-injects the labels.
This results in continuous API server thrashing and the operator logs filling with DEBUG Object updated every few seconds:
DEBUG Object updated {"controller": "psmdb-controller", "name": "mongodb-sample-cluster-cfg-cfg", "kind": "&TypeMeta{Kind:PodDisruptionBudget..."}
DEBUG Object updated {"controller": "psmdb-controller", "name": "mongodb-sample-cluster-mongod-rs0", "kind": "&TypeMeta{Kind:PodDisruptionBudget..."}
Expected Behavior / Proposed Solution:
The Operator already supports the IGNORE_LABELS and IGNORE_ANNOTATIONS environment variables to prevent this exact issue on Service objects and parent CRs.
Please extend the IGNORE_LABELS drift-calculation logic to apply to StatefulSets and PodDisruptionBudgets (at a minimum, PDBs, as they are the most aggressive source of log spam in this scenario).
If a label is specified in IGNORE_LABELS, the Operator should ignore its presence on the STS/PDB and not attempt to strip it during reconciliation.
Current Workarounds:
The only way to stop the API server thrashing is to explicitly exempt the Percona Operator’s service account (or the specific resource kinds it creates) from our Gatekeeper mutation policies. This is a severe compliance issue, as it forces us to run databases without mandatory organizational tracking and monitoring labels.
If there is already an undocumented CRD flag or a better approach to bypass this reconciliation drift for PDBs/STSs, please let me know! Otherwise, extending IGNORE_LABELS would be a massive help for enterprise compliance.
Thanks!