Unable to setup proxy between pmm-client and server

Percona Monitoring and Management (PMM) PMM 2.x
1

Hi there,

I could successfully connect the pmm-client and server without any issues and the metrics are coming like a charm in my grafana dashboard.

After my successful configuration, I tried to set up a proxy server(Nginx) between the pmm client and server. I can set up the basic nginx reverse proxy configuration and able to load the pmm-dashboard URL by using my proxy domain name.

ISSUE:

But the issue popped up when I tried to config the pmm-admin using the proxy domain instead of the direct IP address of my PMM server. It is getting connected and showing in the PMM-Inventory under Nodes, but no metrics showing in the dashboards.

Similiar to the case : https://jira.percona.com/browse/PMM-5368

LOGS/CONFIGS :

Logs from client

Dec 27 07:21:04 server-name pmm-agent: #033[31mERRO#033[0m[2020-12-27T07:21:04.351-05:00] Failed to establish two-way communication channel: context canceled. #033[31mcomponent#033[0m=client

Dec 27 07:21:18 server-name pmm-agent: #033[36mINFO#033[0m[2020-12-27T07:21:18.598-05:00] Connecting to <a href="https://admin:***@proxy-domain.com:443/">https://admin:***@proxy-domain.com:443/</a> ...  #033[36mcomponent#033[0m=client

Dec 27 07:21:18 server-name pmm-agent: #033[36mINFO#033[0m[2020-12-27T07:21:18.615-05:00] Connected to proxy-domain.com:443.     #033[36mcomponent#033[0m=client

Dec 27 07:21:18 server-name pmm-agent: #033[36mINFO#033[0m[2020-12-27T07:21:18.615-05:00] Establishing two-way communication channel ... #033[36mcomponent#033[0m=client

Dec 27 07:21:23 server-name pmm-agent: #033[31mERRO#033[0m[2020-12-27T07:21:23.598-05:00] Failed to establish two-way communication channel: context canceled. #033[31mcomponent#033[0m=client

Logs from proxy server

2020/12/27 07:21:26 [info] 24356#24356: *25170 client canceled stream 1, client: 162.154.189.70, server: proxy-domain.com, request: "POST /agent.Agent/Connect HTTP/2.0", host: "proxy-domain.com"

2020/12/27 07:21:26 [info] 24356#24356: *25170 client prematurely closed stream, client: 162.154.189.70, server: proxy-domain.com, request: "POST /agent.Agent/Connect HTTP/2.0", host: "proxy-domain.com"

2020/12/27 07:21:30 [info] 24356#24356: *25184 client canceled stream 1, client: 162.154.189.182, server: proxy-domain.com, request: "POST /agent.Agent/Connect HTTP/2.0", host: "proxy-domain.com"

2020/12/27 07:21:30 [info] 24356#24356: *25184 client prematurely closed stream, client: 162.154.189.182, server: proxy-domain.com, request: "POST /agent.Agent/Connect HTTP/2.0", host: "proxy-domain.com"

Logs from PMM container(seems like not related):

2020/12/27 12:50:50 [warn] 41#41: *562869 a client request body is buffered to a temporary file /var/cache/nginx/client_temp/0000049734 while sending to client, client: 172.17.0.1, server: _, request: "POST /victoriametrics/api/v1/write HTTP/1.1", host: "pmm-server-IP:443"

2020/12/27 12:50:59 [error] 41#41: *563285 upstream rejected request with error 0 while reading response header from upstream, client: 172.17.0.1, server: _, request: "POST /agent.Agent/Connect HTTP/2.0", upstream: "grpc://127.0.0.1:7771", host: "pmm-server-IP"

2020/12/27 12:51:00 [warn] 41#41: *562869 a client request body is buffered to a temporary file /var/cache/nginx/client_temp/0000049735 while sending to client, client: 172.17.0.1, server: _, request: "POST /victoriametrics/api/v1/write HTTP/1.1", host: "pmm-server-IP:443"

2020/12/27 12:51:02 [error] 41#41: *563289 upstream rejected request with error 0 while reading response header from upstream, client: 172.17.0.1, server: _, request: "POST /agent.Agent/Connect HTTP/2.0", upstream: "grpc://127.0.0.1:7771", host: "pmm-server-IP"

Config I am using in the proxy server:

server {

#listen 80;

listen 443 ssl http2;

server_name proxy-domain.com;

location / {

proxy_pass <a href="https://pmm-server-IP:443/;">https://pmm-server-IP:443/;</a>

proxy_http_version 1.1;

proxy_set_header Upgrade $http_upgrade;

proxy_set_header Connection "upgrade";

proxy_set_header Host $http_host;

proxy_set_header X-Real-IP $remote_addr;

proxy_set_header X-Forward-For $proxy_add_x_forwarded_for;

proxy_set_header X-Forward-Proto http;

proxy_set_header X-Nginx-Proxy true;

proxy_redirect off;

}

  }

Again into the issue:

What I am missing here? Even it is something related to NGINX, could you please give me a hint so that I could work on that. Please note that the dashboard is still accessible under the proxy domain and I can still set up the pmm-client successfully if I am using the IP instead of the proxy domain.

Working:

pmm-admin config --server-insecure-tls --server-url=https://admin:kck1dJpkCD7p5hBHh@pmm-server-IP:443

Not working:

pmm-admin config --server-insecure-tls --server-url=https://admin:kck1dJpkCD7p5hBHh@proxy-domain.com:443

Checking local pmm-agent status…

pmm-agent is running.

Registering pmm-agent on PMM Server...

Registered.

Configuration file /usr/local/percona/pmm2/config/pmm-agent.yaml updated.

Reloading pmm-agent configuration...

Configuration reloaded.

Checking local pmm-agent status...

pmm-agent is running.

 pmm-admin list

Failed to get PMM Server parameters from local pmm-agent: pmm-agent is not connected to PMM Server.

2

Hi, looks like your problem is the same that we have here https://forums.percona.com/discussion/56653/pmm-agent-can-not-connect-to-pmm-server-when-using-reverse-proxy#latest, let’s try to find solution there.

1 Like
3

Hi,
I was working on all the suggestions mentioned in the link you provided. Since I am not that good at Nginx it took me some time to fix the issue. Sharing the details with you.

I have two NGINX proxy servers that are configured with load balancing. All my servers(clients) requests are going through these 2 servers and reach to PMM server(pmm container).

I was able to set up client servers and PMM directly without any issues and the issue was when the requests passing through the proxy-server. I have also configured a proxy domain name for the PMM(All the details are mentioned in my first reply).

Steps I tried.

  1. Tweaked all the firewall rules and make sure that the ports and the IPs are allowed in all servers.
  2. Re-installed the pmm container and assign a custom port for secure connection instead of port 443.
  3. Reconfigured the proxy servers according to the new port
  4. Tried the grpc settings mentioned by you.
  5. Also the SSL settings I tried partially. This means, Installed my domain’s SSL cert on pmm container. Also the same was using on both nginx servers for the proxy domain.(I am not sure these are the steps that the percona techie suggested on the other discussion).

But the issue persisted.

Fix:
Anyway, the fix was related to grpc(or related to all the things I did before). The pmm get connected after I configured the below Nginx conf on my proxy servers by referring to the Nginx documents.

server {
listen 9116 ssl http2;

server_name proxy.mydomain.net;

ssl_certificate /etc/nginx/certificates/wildcard.mydomain.chained.crt;
ssl_certificate_key /etc/nginx/certificates/mydomain.key;
ssl_session_cache shared:SSL:10m;

ssl on;
ssl_session_cache builtin:1000 shared:SSL:10m;
ssl_protocols TLSv1.2;
ssl_ciphers HIGH:!aNULL:!eNULL:!EXPORT:!CAMELLIA:!DES:!MD5:!PSK:!RC4;
ssl_prefer_server_ciphers on;
location / {
grpc_pass grpcs://165.145.588.89:9116;
}
}

References:

Architecture - Percona Monitoring and Management
https://www.percona.com/blog/2020/12/01/foiled-by-the-firewall-a-tale-of-transition-from-prometheus-to-victoriametrics/
Introducing gRPC Support with NGINX 1.13.10 - NGINX
Pmm-agent can not connect to pmm-server when using reverse proxy

4 Likes
4

Hi @jojojoseff
Thank you. It will be very useful for other our users.
We will check it with our QA team and add to our documentation.

1 Like
5

This worked for me, too!

Note that if you use the certificate provided by the container, you have to disable ssl-verfication:

        location / {
            grpc_pass grpcs://172.29.0.108:8080;
            grpc_ssl_verify off;
    }
3 Likes
6

I have been unable to get anything like this to work successfully with apache. Nor have I found a simple setup to use certbot to verify and also use the pmm agent. I really don’t understand why cerbot couldn’t be included in the docker image and used via a setting.

1 Like
7

@nhadsall you don’t need the certs to be present in the container, you can mount the certificate path to the container itself. The certificates are present in /srv/nginx:

$ ls -la /srv/nginx/
total 24
drwxr-xr-x.  2 percona percona  115 Dec  1  2020 .
drwxr-x--x. 12 percona percona  272 Jan 26 14:29 ..
-rw-r--r--.  1 percona percona 6018 Dec  1  2020 ca-certs.pem
-rw-r--r--.  1 percona percona  137 Dec  1  2020 certificate.conf
-rw-r--r--.  1 percona percona  977 Dec  1  2020 certificate.crt
-rw-r--r--.  1 percona percona 1704 Dec  1  2020 certificate.key
-rw-r--r--.  1 percona percona  424 Dec  1  2020 dhparam.pem

So, for example you could use --volume certsvol:/srv/nginx.

At the time any of the certs change, you can either restart the container, or use a less approach such as:

$ docker exec -it pmm-server supervisorctl restart nginx

This then leaves you to provide your certificates however you wish to, including integrating certbot, or similar, outside of the container. In addition, with the certificates outside of the container, you can check validity without needing to connect, such as:

$ openssl x509 -in /path/mounted/to/srv/nginx/certificate.crt -noout -dates

Hopefully, some of that helps you simplify your setup.

1 Like
8

Thanks @Ceri_Williams, the issue isn’t an inability to mount the certs. As I mentioned in my post, the problem is figuring out a way for certbot to authenticate when http traffic is being handled by pmm. That’s the problem I have not been able to solve.

9

Ah, OK you meant specifically for the web request auth. I find that DNS works best, but you need your own DNS for that with a provider that allows you to manage TXT records. That way, you don’t need access and can easily get certs even when the service that would use it is not accessible via the Internet.

1 Like
10

@Ceri_Williams I appreciate the reply. There’s a lot of other considerations here. For various reasons, I’d really just like to figure out a way to have an Apache proxy connections to PMM.

1 Like
11

Since this particular thread is already marked solved, maybe start a new one so we can get into the details of your setup and issue…I’d think this would be great for any future users as well as for the engineering team to better understand the use case and what we might be able to do to address it. I’m kind of seeing two distinct issues…running an intermediate proxy (assuming apache reverse proxy) between PMM clients and PMM server and then managing your certificates via certbot.

1 Like