I am using PMM in docker on aws ec2 instance. Few weeks ago I implement helm chart for deploy PMM in k8s. Chart is pretty simple and works fine in dev environment with ~15 mysql/proxysql instances.
I deployed PMM in PROD and switch all clients to new deployment. It works ok, but not long about 1 hour. PMM server lost all clients. All clients shows:
~# pmm-admin list Failed to get PMM Server parameters from local pmm-agent: pmm-agent is not connected to PMM Server.
And when I try to configure client it shows:
~# pmm-admin config 10.0.0.1 generic master.mysql.db --server-insecure-tls --server-url=http://user:firstname.lastname@example.org/ --force Warning: PMM Server requires TLS communications with client. Checking local pmm-agent status... pmm-agent is running. Registering pmm-agent on PMM Server... Failed to register pmm-agent on PMM Server: response from nginx: <html> <head><title>504 Gateway Time-out</title></head> <body> <center><h1>504 Gateway Time-out</h1></center> <hr><center>nginx</center> </body> </html> . Please check pmm-managed logs..
Grafana on pmm server continue works fine, but Inventory/Settings/Add instance are not response.
Downloaded logs from pmm-server didn’t show any understandable errors for me, but probably I missed sth.
First time I thought that pmm have not enough resources (cpu/memory). No, I give pmm dedicated Node like ec2 it had before.No any concurrency for resources. It didn’t help, it even gave opposite result - faster fail.
Any idea what is going on and how can I realise what I am doing wrong?