Pmm-client does not connect with pmm-server

Percona Monitoring and Management (PMM) PMM 2.x
1

hello everyone,
i’m running pmm-server 2.21.0 on kubernetes environment. I create a nginx ingress and I can login on ppm server with certificate.
I create the user and the role as the documentation said. I already installed pmm-client and connect the client to the server using this command
sudo pmm-admin config --server-insecure-tls --server-url=https://admin:pass@pmm-server.dev.mydomain

and here is where i have problems, because, when i saw the logs from pmm-client:
Sep 21 21:06:42 mongo-dev-n1 pmm-agent[3967094]: ERRO[2021-09-21T21:06:42.739+00:00] Failed to connect to pmm-server.dev.mydomain:443: timeout. component=client

When i connect to pmm-server I can see all the metrics but not the querys. also, when I go to the inventory list I can see the agents there running.

QAN MongoDB Profiler Agent
pmm_agent_id: pmm-server
service_id: /service_id/6d67d21f-861d-443-asddadda
username: pmm-user
tls_skip_verify: true
status: RUNNING
password: ******

can you give me a hand in order to fix this? maybe some annotation is missing in my ingress???

2

Just to be clear…we don’t yet support PMM Server inside Kubernetes…just a standard disclaimer :wink:

I think the issue is that you’re only proxying http(s) traffic through your nginx config. If you can get a terminal in your pmm server and look at the built-in nginx config you’ll see we also use gRPC encapsulated in https. Here’s a few of them (not complete list) but you can look in /etc/nginx/conf.d/pmm.conf to see everything:

    location /agent. {
      grpc_pass grpc://managed-grpc;
      # Disable request body size check for gRPC streaming, see https://trac.nginx.org/nginx/ticket/1642.
      # pmm-managed uses grpc.MaxRecvMsgSize for that.
      client_max_body_size 0;
    }
    location /inventory. {
      grpc_pass grpc://managed-grpc;
    }
    location /management. {
      grpc_pass grpc://managed-grpc;
    }
    location /server. {
      grpc_pass grpc://managed-grpc;
    }

So your timeout error (vs a connection refused) is because we spoke gRPC and got no response.

1 Like
3

hello steve! thank you so much for your time and help!

I will check what you told me. There is a way to test (using curl o something) if the ingress is the problem? what path should I test in order to make the connection pmm-client <–>pmm-server?

I can create a ILB, but maybe it’s not the best. I mean, have one for only one service.

1 Like
4

I don’t think so but will let someone smarter than me chime in on that. I think all our API endpoints are http and not gRPC. My guess on a fix is you can simply create a single entry to take all gRPC traffic to your nginx and route it to the gRPC of your pmm server (where there’s another nginx running).

Is there a reason you prefer to run your PMM server inside Kubernetes? if it’s to monitor db’s inside kubernetes we can do that with PMM client now that push metrics are our default.

1 Like
5

steve thanks again. i have fixed the problem creating internal load balancer. it was the only way. i tried a lot of things, like using grcp backend annotation on nginx ingress or adding some of the path you told me before with no success.

At least I understand the problem now. you can close this topic now. Thanks.

1 Like