Register and ask your own question!
Memory management and RAM OOM kill for PXC pods
In our company, we are using Percona XtraDB Cluster operator to provide one MySQL cluster for each one of our customers due to internal reasons not worth diving into right now.
As our tests approach the final stages of development, we set Kubernetes resource limits to cap the RAM each PXC pod would be able to use but we had to remove this specific configuration because it appears the PXC pods can't/won't respect any memory boundaries. Once an OOM kills any PXC pod, usually in the middle of a regular workload, the cluster can't come back online on it's own and is precisely why we just turned the limit off for the time being.
As our undestanding of RAM usage in MySQL is limited and PXC also deploys the Galera replication, not to mention the RAM management done by Kubernetes could be a factor here, we are reaching out to see what could be done to enforce the resources as some configuration to
mysqld and/or other programs so they never even allocate more RAM than they should, avoiding any OOM kill in the first place.
Currently, we are using
my.cnf to configure the PXC pod, but to no avail. Any time we try to restore a database with huge INSERT INTO queries or other operations like that the pod inevitably allocates however much memory it wants to... Also, once allocated, the pods never deallocate it, hence possibly being a Kubernetes behavior after some memory spike.
This is a huge problem since, as there will be a whole lot of clusters in our environment, we need to set a reasonably low memory usage to each pod as CPU/RAM is the bulk of our cost structure. Even considering the memory limit is an arbitrary value, the fact that we can't even set 1GB as per the example is the problem we are trying to figure a solution for.
Regards from Brazil, DAVI