Unable to setup proxy between pmm-client and server

Hi there,

I could successfully connect the pmm-client and server without any issues and the metrics are coming like a charm in my grafana dashboard.

After my successful configuration, I tried to set up a proxy server(Nginx) between the pmm client and server. I can set up the basic nginx reverse proxy configuration and able to load the pmm-dashboard URL by using my proxy domain name.

ISSUE:

But the issue popped up when I tried to config the pmm-admin using the proxy domain instead of the direct IP address of my PMM server. It is getting connected and showing in the PMM-Inventory under Nodes, but no metrics showing in the dashboards.

Similiar to the case : https://jira.percona.com/browse/PMM-5368

LOGS/CONFIGS :

Logs from client

Dec 27 07:21:04 server-name pmm-agent: #033[31mERRO#033[0m[2020-12-27T07:21:04.351-05:00] Failed to establish two-way communication channel: context canceled. #033[31mcomponent#033[0m=client

Dec 27 07:21:18 server-name pmm-agent: #033[36mINFO#033[0m[2020-12-27T07:21:18.598-05:00] Connecting to <a href="https://admin:***@proxy-domain.com:443/">https://admin:***@proxy-domain.com:443/</a> ...  #033[36mcomponent#033[0m=client

Dec 27 07:21:18 server-name pmm-agent: #033[36mINFO#033[0m[2020-12-27T07:21:18.615-05:00] Connected to proxy-domain.com:443.     #033[36mcomponent#033[0m=client

Dec 27 07:21:18 server-name pmm-agent: #033[36mINFO#033[0m[2020-12-27T07:21:18.615-05:00] Establishing two-way communication channel ... #033[36mcomponent#033[0m=client

Dec 27 07:21:23 server-name pmm-agent: #033[31mERRO#033[0m[2020-12-27T07:21:23.598-05:00] Failed to establish two-way communication channel: context canceled. #033[31mcomponent#033[0m=client

Logs from proxy server

2020/12/27 07:21:26 [info] 24356#24356: *25170 client canceled stream 1, client: 162.154.189.70, server: proxy-domain.com, request: "POST /agent.Agent/Connect HTTP/2.0", host: "proxy-domain.com"

2020/12/27 07:21:26 [info] 24356#24356: *25170 client prematurely closed stream, client: 162.154.189.70, server: proxy-domain.com, request: "POST /agent.Agent/Connect HTTP/2.0", host: "proxy-domain.com"

2020/12/27 07:21:30 [info] 24356#24356: *25184 client canceled stream 1, client: 162.154.189.182, server: proxy-domain.com, request: "POST /agent.Agent/Connect HTTP/2.0", host: "proxy-domain.com"

2020/12/27 07:21:30 [info] 24356#24356: *25184 client prematurely closed stream, client: 162.154.189.182, server: proxy-domain.com, request: "POST /agent.Agent/Connect HTTP/2.0", host: "proxy-domain.com"

Logs from PMM container(seems like not related):

2020/12/27 12:50:50 [warn] 41#41: *562869 a client request body is buffered to a temporary file /var/cache/nginx/client_temp/0000049734 while sending to client, client: 172.17.0.1, server: _, request: "POST /victoriametrics/api/v1/write HTTP/1.1", host: "pmm-server-IP:443"

2020/12/27 12:50:59 [error] 41#41: *563285 upstream rejected request with error 0 while reading response header from upstream, client: 172.17.0.1, server: _, request: "POST /agent.Agent/Connect HTTP/2.0", upstream: "grpc://127.0.0.1:7771", host: "pmm-server-IP"

2020/12/27 12:51:00 [warn] 41#41: *562869 a client request body is buffered to a temporary file /var/cache/nginx/client_temp/0000049735 while sending to client, client: 172.17.0.1, server: _, request: "POST /victoriametrics/api/v1/write HTTP/1.1", host: "pmm-server-IP:443"

2020/12/27 12:51:02 [error] 41#41: *563289 upstream rejected request with error 0 while reading response header from upstream, client: 172.17.0.1, server: _, request: "POST /agent.Agent/Connect HTTP/2.0", upstream: "grpc://127.0.0.1:7771", host: "pmm-server-IP"

Config I am using in the proxy server:

server {

#listen 80;

listen 443 ssl http2;

server_name proxy-domain.com;

location / {

proxy_pass <a href="https://pmm-server-IP:443/;">https://pmm-server-IP:443/;</a>

proxy_http_version 1.1;

proxy_set_header Upgrade $http_upgrade;

proxy_set_header Connection "upgrade";

proxy_set_header Host $http_host;

proxy_set_header X-Real-IP $remote_addr;

proxy_set_header X-Forward-For $proxy_add_x_forwarded_for;

proxy_set_header X-Forward-Proto http;

proxy_set_header X-Nginx-Proxy true;

proxy_redirect off;

}

  }

Again into the issue:

What I am missing here? Even it is something related to NGINX, could you please give me a hint so that I could work on that. Please note that the dashboard is still accessible under the proxy domain and I can still set up the pmm-client successfully if I am using the IP instead of the proxy domain.

Working:

pmm-admin config --server-insecure-tls --server-url=https://admin:kck1dJpkCD7p5hBHh@pmm-server-IP:443

Not working:

pmm-admin config --server-insecure-tls --server-url=https://admin:kck1dJpkCD7p5hBHh@proxy-domain.com:443

Checking local pmm-agent status…

pmm-agent is running.

Registering pmm-agent on PMM Server...

Registered.

Configuration file /usr/local/percona/pmm2/config/pmm-agent.yaml updated.

Reloading pmm-agent configuration...

Configuration reloaded.

Checking local pmm-agent status...

pmm-agent is running.

 pmm-admin list

Failed to get PMM Server parameters from local pmm-agent: pmm-agent is not connected to PMM Server.

Hi, looks like your problem is the same that we have here https://forums.percona.com/discussion/56653/pmm-agent-can-not-connect-to-pmm-server-when-using-reverse-proxy#latest, let’s try to find solution there.

1 Like

Hi,
I was working on all the suggestions mentioned in the link you provided. Since I am not that good at Nginx it took me some time to fix the issue. Sharing the details with you.

I have two NGINX proxy servers that are configured with load balancing. All my servers(clients) requests are going through these 2 servers and reach to PMM server(pmm container).

I was able to set up client servers and PMM directly without any issues and the issue was when the requests passing through the proxy-server. I have also configured a proxy domain name for the PMM(All the details are mentioned in my first reply).

Steps I tried.

  1. Tweaked all the firewall rules and make sure that the ports and the IPs are allowed in all servers.
  2. Re-installed the pmm container and assign a custom port for secure connection instead of port 443.
  3. Reconfigured the proxy servers according to the new port
  4. Tried the grpc settings mentioned by you.
  5. Also the SSL settings I tried partially. This means, Installed my domain’s SSL cert on pmm container. Also the same was using on both nginx servers for the proxy domain.(I am not sure these are the steps that the percona techie suggested on the other discussion).

But the issue persisted.

Fix:
Anyway, the fix was related to grpc(or related to all the things I did before). The pmm get connected after I configured the below Nginx conf on my proxy servers by referring to the Nginx documents.

server {
listen 9116 ssl http2;

server_name proxy.mydomain.net;

ssl_certificate /etc/nginx/certificates/wildcard.mydomain.chained.crt;
ssl_certificate_key /etc/nginx/certificates/mydomain.key;
ssl_session_cache shared:SSL:10m;

ssl on;
ssl_session_cache builtin:1000 shared:SSL:10m;
ssl_protocols TLSv1.2;
ssl_ciphers HIGH:!aNULL:!eNULL:!EXPORT:!CAMELLIA:!DES:!MD5:!PSK:!RC4;
ssl_prefer_server_ciphers on;
location / {
grpc_pass grpcs://165.145.588.89:9116;
}
}

References:

Percona Monitoring and Management
Foiled by the Firewall: A Tale of Transition From Prometheus to VictoriaMetrics - Percona Database Performance Blog
https://www.nginx.com/blog/nginx-1-13-10-grpc/
Pmm-agent can not connect to pmm-server when using reverse proxy

2 Likes

Hi @jojojoseff
Thank you. It will be very useful for other our users.
We will check it with our QA team and add to our documentation.

1 Like

This worked for me, too!

Note that if you use the certificate provided by the container, you have to disable ssl-verfication:

        location / {
            grpc_pass grpcs://172.29.0.108:8080;
            grpc_ssl_verify off;
    }
2 Likes