External access to PMM prometheus in pmm 2.10.1

Percona Monitoring and Management (PMM) PMM 2.x
1

Hello.

In new pmm release 2.10.1 - pmm reject access to prometheus throw port 9090.

I have my own Grafana server where I built some custom dashboards for alerting (yes, it was before alert manager became to pmm). Grafana is using pmm prometheus as a source like http://pmm.prod.com:9090/prometheus/

~# curl http://pmm.prod.com:9090/prometheus/
<a href="/prometheus/graph">Found</a>.

In 2.10.1 9090 port is closed, or sth like it. Why? And how to open it as it was before

~# curl http://pmm.prod.com:9090/prometheus/
curl: (7) Failed to connect to pmm.prod.com port 9090: Connection refused

From deployment perspective nothing changed from my side:

docker run -d -p 80:80 -p 443:443 -p 9090:9090 --volumes-from pmm-data --name pmm-server --restart always -e GF_AUTH_LDAP_CONFIG_FILE=ldap.toml -e GF_AUTH_LDAP_ENABLED=true percona/pmm-server:2
2

Hi,

You may allow access to prometheus location for your own grafana instance in file /etc/nginx/conf.d/pmm.conf

Next lines should be added

satisfy any;

 allow 1.2.3.4;

 deny all;

e.g.

Screenshot_20200923_122459.png

3

thanks for information, but it didn’t help.

I have update /etc/nginx/conf.d/pmm.conf file

# Prometheus

location /prometheus {

satisfy any;

allow 172.xxx.xxx.xxx;

deny all;

proxy_pass http://127.0.0.1:9090;

proxy_read_timeout 600;

}

Restart docker container. But the problem still exists. Grafana shows:

data:""

status:502

statusText:"Bad Gateway"

xhrStatus:"complete"

request:Object

method:"GET"

url:"api/datasources/proxy/16/api/v1/query_range....
4

thanks for update, but it didn’t help.

I have updated pmm once again, update /etc/nginx/conf.d/pmm.conf file:

# Prometheus

location /prometheus {

satisfy any;

allow 172.xxx.xxx.xxx;

deny all;

proxy_pass http://127.0.0.1:9090;

proxy_read_timeout 600;

}

restart docker container. PMM started and works ok, but from my Grafana perspective nothing changed. Here is example of Query Inspector output:

data:""

status:502

statusText:"Bad Gateway"

xhrStatus:"complete"

request:Object

method:"GET"

url:"api/datasources/proxy/16/api/v1/query_range...
5

Could you test it with parameter “allow all”?

6

unfortunately not.

# Prometheus
    location /prometheus {
      satisfy any;
      allow 172.xxx.xxx.xxx;
      allow all;
      proxy_pass http://127.0.0.1:9090;
      proxy_read_timeout 600;
    }
 # Prometheus
    location /prometheus {
      satisfy any;
      allow all;
      proxy_pass http://127.0.0.1:9090;
      proxy_read_timeout 600;
    }

both tries gave the same result: Grafana shows error 502 - Bad Gateway.

7

Are you still trying connect to Prometheus directly? I mean port 9090.

Suggested changes allow to access Prometheus through nginx proxying.

So please try command

curl http://pmm.prod.com/prometheus/
8

@adivinhohuge thanks for help and patience. My final config looks like:

# Prometheus
    location /prometheus {
      satisfy any;
      allow 172.xxx.xxx.xxx;
      deny all;
      proxy_pass http://127.0.0.1:9090;
      proxy_read_timeout 600;
    }

I have range of ips for like 172.0.0.0/21…

And Grafana datasource looks like

url: http://pmm.prod..com/prometheus/

instead of

url: http://pmm.prod..com:9090/prometheus/

Also I changed a little security groups because Grafana starts using another port to connect pmm-server.

Everything works now. Thanks again.