Pmm client getting OOM after few days in everest deployed cluster

Percona Everest
,
1

I am running a PXC 3 node cluster using everest. After a few days I am getting this error.

terminated
Reason:Reason: OOMKilled - exit code: 137

for the pmm-client container alone and this is happening for the first node alone.
Any help on how to address this or can someone point me to how to debug this thing.

2

OOM means there is not enough memory on that node. Can you increase the amount of memory? Use PMM to track memory usage and see which process is using too much.

3

This is a 32 gb instance. and there are no other significant pods scheduled in the node except for this 4 cpu 7gb pxc-0 db.

I did check here pmm sharing the screen shots

image
image1580×603 76.8 KB

image
image1519×427 42.6 KB

My question here would be if resources for pmm client alone can be increased?
Also i have no idea why the pxc-0 memory is hitting the limit so quickly either. i was doing some imports to tables but as you can see storage is just about 10gb so it is not a big database.
Later i turned off the pmm monitoring since the restarts keep happening. when i turn it back on it keeps happening again

image
image1493×384 34.8 KB

Previously i got the same issue i uninstalled everest and did full install again it was fine for a week and then this happens after some data load of few gb.

4

Hi @beta-auction ,

Can you share:

  • K8s platform and version
  • Everest version
  • PMM server version
  • PXC version

How do you create a cluster in Everest, I mean the resources and nodes and do you set some custom database settings?

Thank you!

5

I am using RKE2 cluster, 1.30.2.
Everest used 1.1.0 (updated to 1.1.1 and still the same issue after turning monitoring off and then on)
PMM is 2.42 ( latest version)
PXC Version: 8.0.36-28.1 ( latest available in everest)

Cluster is created on aws, db nodes are are 3 ( all are m6a.2xlarge ). The percona cluster is 3 node medium config as specified in everest ui.

The db settings i have as

[mysqld]
max_connections = 2000
thread_cache_size = 48
thread_pool_size = 8
table_open_cache_instances = 8
wait_timeout = 900
interactive_timeout = 900
default_time_zone = '+00:00'

Apart from usual db operation for my apps, i had done few imports(all less than few gb). And as you can see in the pics above max db size less 10 gb. Few tables have around million rows.