Apache: Unable to setup proxy between pmm-client and server

Percona Monitoring and Management (PMM) PMM 2.x
1

I can easily setup a h2 proxy and get it to work via a browser. The agent will connect, but seems that grpc just won’t work.

1 Like
2

Ok, starting with errors, what are you seeing for the pmm-agent (syslog)?

The agent to server connection needs to be always up so keepalives would be the first place I’d look since you’d need to broker these connections you can’t have agent to apache up, apache to nginx closed.

And it was mentioned in the other thread but I assume the custom cert you’re using is valid and trusted by apache…there’s no errors in the apache logs about the cert from the PMM server not passing?

1 Like
3

Thanks so much @steve.hoffman!

Sorry, I take that back. I was fighting with it again late last night and remembered incorrectly.

I am able to get an https proxy working, but once I use h2, nginx is throwing a 400, “Bad Request”

	RewriteEngine On
	SSLProxyEngine on
	ProxyPreserveHost off
	ProxyRequests on
	SSLProxyCheckPeerName off
	SSLProxyVerify none
	SSLProxyCheckPeerCN off
	SSLProxyCheckPeerExpire off
	AllowEncodedSlashes On

#	RewriteCond %{HTTP:Content-Type} =application/grpc [NC]
#	RewriteRule /(.*) h2://10.0.0.50:2443/$1 [P,L]
#	RewriteRule /(.*) https://10.0.0.50:2443/$1 [P,L]

	ProxyPass / h2://10.0.0.50:2443/
	ProxyPassReverse / https://10.0.0.50:2443/

Currently I’m using ProxyPass & ProxyPassReverse, but previously I also used RewriteRule.

I’ve been able to curl to docker from the proxy’s server using --htt2, so that seems to be working.

1 Like
4

Scratching my head here…

I assume you’ve enabled mod_proxy_http2 (I guess the issue would be the proxyPass h2:// couldn’t be understood if it weren’t)

My theories are mixing and matching of protocols (http 1 vs 2 which apache says it won’t downgrade) but doesn’t seem to be the case where chrome makes http1.1 request so maybe check with whatever browser/tool you’re using to get the 400 error.

Connection shutting down somehwere…but if you’re saying nginx is what’s returning the 400 (I’m assuming browser message vs nginx logs on pmm-server) that’s unlikely to be the case.

So I guess my next question would be: with the grpc proxyPass statement configured is anything working?

  • Can you go to a browser and put in your apache proxy’s https url and get the UI to show up? I assume the UI works fine if you go right to the nginx interface.
  • Looking at https://:2443/victoriametrics/targets are the clients reporting healthy or unhealthy?
  • Is data from the exporters making it to PMM? (would say gRPC is working but https is not)

Sorry I’m not much help but short of putting wireshark on the apache server to see what’s coming into the proxy vs going out to the nginx I’m not sure where else to look.

1 Like