Add Blackbox exporter as external service?

Percona Monitoring and Management (PMM) PMM 2.x
1

I am trying to set up Blackbox Exporter for PMM, but adding it as an external service unfortunately fails:

pmm-admin add external \
               --metrics-path='probe?target=google.com&module=icmp' \
               --service-name="gen-test-1-blackbox_icmp.google_com" \
               --listen-port=42003 \
               --custom-labels=blackbox_target='google.com',blackbox_module='icmp' \
               --scheme=http \
               --environment=test

This returns
Connection check failed: Unexpected exporter's response format: text format parsing error in line 1: invalid metric name.
Running pmm-admin with --debug shows this output (besides some other stuff):

{"runs_on_node_id":"/node_id/731a4bf7-65df-4e33-b662-cfc15960c342","service_name":"gen-test-1-blackbox_icmp.google_com","scheme":"http","metrics_path":"/probe?target=google.com&module=icmp","listen_port":42003,"node_id":"/node_id/731a4bf7-65df-4e33-b662-cfc15960c342","environment":"test","custom_labels":{"blackbox_module":"icmp","blackbox_target":"google.com"},"group":"external","metrics_mode":"AUTO"}

DEBUG 2023-04-14 16:12:49.554852369Z: HTTP/1.1 400 Bad Request
Content-Length: 292
Connection: keep-alive
Content-Type: application/json
Date: Fri, 14 Apr 2023 16:12:49 GMT
Server: nginx
Strict-Transport-Security: max-age=63072000; includeSubdomains;

{
  "error":  "Connection check failed: Unexpected exporter's response format: text format parsing error in line 1: invalid metric name.",
  "code":  9,
  "message":  "Connection check failed: Unexpected exporter's response format: text format parsing error in line 1: invalid metric name."
}
DEBUG 2023-04-14 16:12:49.558287917Z: Result: <nil>
DEBUG 2023-04-14 16:12:49.558666304Z: Error: &external.AddExternalDefault{_statusCode:400, Payload:(*external.AddExternalDefaultBody)(0xc0002ba090)}
Connection check failed: Unexpected exporter's response format: text format parsing error in line 1: invalid metric name.

As blackbox exporter is definitely working (icmp module is configured, and curl ‘127.0.0.1:42003/probe?target=google.com&module=icmp’ does return a valid response) I assume that either my pmm admin command is wrong or the metrics path is being sanitized and the parameters (?target=google.com&module=icmp) are removed.

This is probably not the recommended way of probing blackbox exporter anyway, so how to properly use blackbox exporter with PMM (besides creating files for the textfile collector)?

Thanks!
Rainer

2

You have verified, via curl, that the exporter URL is correct? It looks like it should be:
http://localhost:42003/probe?target=google.com&module=icmp

Does that work when you curl it? Does it give a proper prometheus output?

3

Yes. I previously only checked with 127.0.0.1:42003/probe?target=google.com&module=icmp, but http://localhost:42003/probe?target=google.com&module=icmp does work as well.

4

Can you show the output of your attempt using curl? PMM says the output is invalid. Show the full command you are testing and full output please.

5 
> curl 'http://localhost:42003/probe?target=google.com&module=icmp'
# HELP probe_dns_lookup_time_seconds Returns the time taken for probe dns lookup in seconds
# TYPE probe_dns_lookup_time_seconds gauge
probe_dns_lookup_time_seconds 0.002958326
# HELP probe_duration_seconds Returns how long the probe took to complete in seconds
# TYPE probe_duration_seconds gauge
probe_duration_seconds 0.009676323
# HELP probe_icmp_duration_seconds Duration of icmp request by phase
# TYPE probe_icmp_duration_seconds gauge
probe_icmp_duration_seconds{phase="resolve"} 0.002958326
probe_icmp_duration_seconds{phase="rtt"} 0.005955353
probe_icmp_duration_seconds{phase="setup"} 0.000206063
# HELP probe_icmp_reply_hop_limit Replied packet hop limit (TTL for ipv4)
# TYPE probe_icmp_reply_hop_limit gauge
probe_icmp_reply_hop_limit 116
# HELP probe_ip_addr_hash Specifies the hash of IP address. It's useful to detect if the IP address changes.
# TYPE probe_ip_addr_hash gauge
probe_ip_addr_hash 4.015196223e+09
# HELP probe_ip_protocol Specifies whether probe ip protocol is IP4 or IP6
# TYPE probe_ip_protocol gauge
probe_ip_protocol 4
# HELP probe_success Displays whether or not the probe was a success
# TYPE probe_success gauge
probe_success 1
6

That looks good. When you add the external to PMM, try escaping the & Check the PMM logs as well to see what exactly it is trying to load.

7

So most certainly it is the query in metrics-path that gets escaped by go:

u.Path = "probe?target=google.com&module=icmp"
fmt.Println(u)

out:
probe%3Ftarget=google.com&module=icmp

In our current implementation metrics-path directly passes to the URL.Path :

Which then gets escaped by URI encoding rules for the Path field.

It looks like there is no way to escape it.

I have created [PMM-12043] Not possible to add external service with metrics-path that includes query - Percona JIRA

8

Just wanted to play around with escaping, so this saved me some time, thanks for that!

This leaves me with one question though:
Setting up an external service is sufficient in order to use blackbox exporter for a couple of probes; but for a more elaborate setup (e.g. for monitoring several dozens of IP addresses or web sites) this would not be useful. I did not find a way to add an additional scrape configuration to PMM in order to utilize the “multi-target” export pattern as described for Blackbox Exporter, which also would allow relabeling.
It seems (or seemed) to be possible by manually editing the server’s configuration (see e.g. [PMM-5812] Prometheus relabeling broken (relabel_configs unmarshal errors) - Percona JIRA), but this does not look like to be “officially supported” - or would this be the way to go?

9

@rapli it is supported: FAQ - Percona Monitoring and Management

But it is broken again: [PMM-11196] pmm-managed failed to update configuration - Percona JIRA

10

@rapli, you are right. There should be a way to have a query for the metric endpoint. It could be used for multi-target or to filter results (GitHub - prometheus/node_exporter: Exporter for machine metrics).

I would agree that, in your case, custom prom config is a way to go. I hope it will be fixed soon.