Process-exporter and PMM

Percona Monitoring and Management (PMM) PMM 2.x
1

Description:

Unable to add External process Monitor

Steps to Reproduce:

Install pmm and standard YUM process that works properly as well as mysql monitor All agents and items are being monitored properly.

Install of process-exporter

  1. Reason: I want to be able to monitor the amount of items like httpd processes and or others and it appears that with the default pmm we can not (I want to generate alerts if X happens and do not want to add tons of cronjobs to the text exporter)
wget -q https://github.com/ncabatoff/process-exporter/releases/download/v0.7.10/process-exporter-0.7.10.linux-amd64.tar.gz
tar xvfz process-exporter-0.7.10.linux-amd64.tar.gz
cd process-exporter-*
mv process-exporter /usr/local/bin/
cd ..
rm -rf process-exporter*

cat << EOF > /etc/systemd/system/process-exporter.service
[Unit]
Description=Promtail service
After=network.target

[Service]
Type=simple
User=root
ExecStart=/usr/local/bin/process-exporter -config.path /etc/process-exporter.yml

[Install]
WantedBy=multi-user.target
EOF

systemctl enable process-exporter.service
systemctl restart process-exporter.service
pmm-admin add external --username=xx --password=xx --listen-port=9256

All desired metrics are being published at the 9256 port on the server that I want.

pmm-admin add external --help

lists the proper values such as service name agent (auto-detect)

pmm-server under inventory shows Failed and Status NA

Version:

Version: 2.38.1
PMMVersion: 2.38.1

**Note Server is 2.38 as I have not updated the server yet but as it is a security fix only I would expect I would be ok

Logs:

 pmm-admin add external --username=xxxx --password=xxxx --listen-port=9256 --trace
DEBUG 2023-07-16 20:43:18.737413349Z: /tmp/go/pkg/mod/github.com/go-openapi/runtime@v0.26.0/client/runtime.go:450 github.com/go-openapi/runtime/client.(*Runtime).Submit() POST /local/Status HTTP/1.1
Host: 127.0.0.1:7777
User-Agent: Go-http-client/1.1
Content-Length: 3
Accept: application/json
Content-Type: application/json
Accept-Encoding: gzip

{}

DEBUG 2023-07-16 20:43:18.738451233Z: /tmp/go/pkg/mod/github.com/go-openapi/runtime@v0.26.0/client/runtime.go:497 github.com/go-openapi/runtime/client.(*Runtime).Submit() HTTP/1.1 200 OK
Content-Length: 1635
Content-Type: application/json
Date: Sun, 16 Jul 2023 20:43:18 GMT
Grpc-Metadata-Content-Type: application/grpc

{
  "agent_id": "/agent_id/d64707a1-76bf-410e-988e-7670cd1c0986",
  "runs_on_node_id": "/node_id/1b1b3396-1a5a-4ed4-a6c9-d0266763459b",
  "server_info": {
    "url": "https://api_key:e---api-key----MjU0MiIsImlkIjoxfQ==@zzzzz.com:443/",
    "insecure_tls": true,
    "connected": true,
    "version": "2.38.0",
    "latency": null,
    "clock_drift": null
  },
  "agents_info": [
    {
      "agent_id": "/agent_id/34915748-f4bb-466f-86fc-132438a91415",
      "agent_type": "NODE_EXPORTER",
      "status": "RUNNING",
      "listen_port": 42000,
      "process_exec_path": "/usr/local/percona/pmm2/exporters/node_exporter"
    },
    {
      "agent_id": "/agent_id/5510d5c7-7e60-4791-ac9a-32b3dac51e73",
      "agent_type": "QAN_MYSQL_SLOWLOG_AGENT",
      "status": "RUNNING",
      "listen_port": 0,
      "process_exec_path": ""
    },
    {
      "agent_id": "/agent_id/c948d04c-d95b-4a4b-aae8-1957e16495e2",
      "agent_type": "MYSQLD_EXPORTER",
      "status": "RUNNING",
      "listen_port": 42004,
      "process_exec_path": "/usr/local/percona/pmm2/exporters/mysqld_exporter"
    },
    {
      "agent_id": "/agent_id/d8c92aad-2d1a-47ee-b5f5-4737db29d2f1",
      "agent_type": "VM_AGENT",
      "status": "RUNNING",
      "listen_port": 42001,
      "process_exec_path": "/usr/local/percona/pmm2/exporters/vmagent"
    }
  ],
  "config_filepath": "/usr/local/percona/pmm2/config/pmm-agent.yaml",
  "agent_version": "2.38.1",
  "node_name": "xxxx.com",
  "connection_uptime": 100
}
DEBUG 2023-07-16 20:43:18.738907186Z: /tmp/go/pkg/mod/github.com/go-openapi/runtime@v0.26.0/client/runtime.go:450 github.com/go-openapi/runtime/client.(*Runtime).Submit() POST /local/Status HTTP/1.1
Host: 127.0.0.1:7777
User-Agent: Go-http-client/1.1
Content-Length: 3
Accept: application/json
Content-Type: application/json
Accept-Encoding: gzip

{}

DEBUG 2023-07-16 20:43:18.73942799Z: /tmp/go/pkg/mod/github.com/go-openapi/runtime@v0.26.0/client/runtime.go:497 github.com/go-openapi/runtime/client.(*Runtime).Submit() HTTP/1.1 200 OK
Content-Length: 1635
Content-Type: application/json
Date: Sun, 16 Jul 2023 20:43:18 GMT
Grpc-Metadata-Content-Type: application/grpc

{
  "agent_id": "/agent_id/d64707a1-76bf-410e-988e-7670cd1c0986",
  "runs_on_node_id": "/node_id/1b1b3396-1a5a-4ed4-a6c9-d0266763459b",
  "server_info": {
    "url": "https://api_key:e---api-key----MjU0MiIsImlkIjoxfQ==@zzzzz.com:443/",
    "insecure_tls": true,
    "connected": true,
    "version": "2.38.0",
    "latency": null,
    "clock_drift": null
  },
  "agents_info": [
    {
      "agent_id": "/agent_id/34915748-f4bb-466f-86fc-132438a91415",
      "agent_type": "NODE_EXPORTER",
      "status": "RUNNING",
      "listen_port": 42000,
      "process_exec_path": "/usr/local/percona/pmm2/exporters/node_exporter"
    },
    {
      "agent_id": "/agent_id/5510d5c7-7e60-4791-ac9a-32b3dac51e73",
      "agent_type": "QAN_MYSQL_SLOWLOG_AGENT",
      "status": "RUNNING",
      "listen_port": 0,
      "process_exec_path": ""
    },
    {
      "agent_id": "/agent_id/c948d04c-d95b-4a4b-aae8-1957e16495e2",
      "agent_type": "MYSQLD_EXPORTER",
      "status": "RUNNING",
      "listen_port": 42004,
      "process_exec_path": "/usr/local/percona/pmm2/exporters/mysqld_exporter"
    },
    {
      "agent_id": "/agent_id/d8c92aad-2d1a-47ee-b5f5-4737db29d2f1",
      "agent_type": "VM_AGENT",
      "status": "RUNNING",
      "listen_port": 42001,
      "process_exec_path": "/usr/local/percona/pmm2/exporters/vmagent"
    }
  ],
  "config_filepath": "/usr/local/percona/pmm2/config/pmm-agent.yaml",
  "agent_version": "2.38.1",
  "node_name": "xxxx.com",
  "connection_uptime": 100
}
DEBUG 2023-07-16 20:43:18.739762186Z: /tmp/go/pkg/mod/github.com/go-openapi/runtime@v0.26.0/client/runtime.go:450 github.com/go-openapi/runtime/client.(*Runtime).Submit() POST /v1/management/External/Add HTTP/1.1
Host: zzzzz.com:443
User-Agent: Go-http-client/1.1
Content-Length: 277
Accept: application/json
Authorization: Basic YXBpX2tleTpleUpySWpvaVEzbHlNRFF3UVdrMVkyeFFVR0ZRVGtOM00zZDRWRnBUVFhWWWFsUnhlbVlpTENKdUlqb2ljRzF0TFdGblpXNTBMWFJ3WVd4d1luUXhkbUV3TVRJdWRtVnlhWHB2Ymk1amIyMHROelEyT0RRM09UWXdNelF4TnpNNU1qVTBNaUlzSW1sa0lqb3hmUT09
Content-Type: application/json
Accept-Encoding: gzip

{"runs_on_node_id":"/node_id/1b1b3396-1a5a-4ed4-a6c9-d0266763459b","service_name":"xxxx.com-external","username":"xxxx","password":"xxxx","listen_port":9256,"node_id":"/node_id/1b1b3396-1a5a-4ed4-a6c9-d0266763459b","group":"external","metrics_mode":"AUTO"}

DEBUG 2023-07-16 20:43:19.162040791Z: /tmp/go/pkg/mod/github.com/go-openapi/runtime@v0.26.0/client/runtime.go:497 github.com/go-openapi/runtime/client.(*Runtime).Submit() HTTP/1.1 200 OK
Content-Length: 613
Cache-Control: no-cache
Connection: keep-alive
Content-Type: application/json
Date: Sun, 16 Jul 2023 20:43:19 GMT
Grpc-Metadata-Content-Type: application/grpc
Pragma: no-cache
Server: nginx
Strict-Transport-Security: max-age=63072000; includeSubdomains;
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
X-Xss-Protection: 1; mode=block

{
  "service": {
    "service_id": "/service_id/dbe4d916-7d04-4ebd-8c20-ed2468776fc8",
    "service_name": "xxxx.com-external",
    "node_id": "/node_id/1b1b3396-1a5a-4ed4-a6c9-d0266763459b",
    "group": "external"
  },
  "external_exporter": {
    "agent_id": "/agent_id/461ec5ba-3d68-41d4-80d7-842b9249c65c",
    "runs_on_node_id": "/node_id/1b1b3396-1a5a-4ed4-a6c9-d0266763459b",
    "service_id": "/service_id/dbe4d916-7d04-4ebd-8c20-ed2468776fc8",
    "username": "xxxx",
    "scheme": "http",
    "metrics_path": "/metrics",
    "listen_port": 9256,
    "push_metrics_enabled": true
  }
}
DEBUG 2023-07-16 20:43:19.162212329Z: /tmp/go/src/github.com/percona/pmm/admin/cli/cli.go:130 github.com/percona/pmm/admin/cli.printResponse() Result: &management.addExternalResult{Service:(*external.AddExternalOKBodyService)(0xc000268100)}
DEBUG 2023-07-16 20:43:19.162230247Z: /tmp/go/src/github.com/percona/pmm/admin/cli/cli.go:131 github.com/percona/pmm/admin/cli.printResponse() Error: <nil>
External Service added.
Service ID  : /service_id/dbe4d916-7d04-4ebd-8c20-ed2468776fc8
Service name: xxxx.com-external
Group       : external

To Date I have yet to be able to figure out what I am doing incorrectly… and tried a dozen or more different options… Can someone give me a hit of a bit of help?

***Note I suspect it is with the following but when I try to add this in the same format or similar format that I do for the initial setup pmm-admin config --server-insecure-tls --server-url=http://xx:xx@xx.xx.com:443

pmm-admin add external --username=xx --password=xx --listen-port=9256 --server-url=http://xx.xx.com

gives me a nginx 404 page not found error…

Thanks

2

Hi @meyerder

Could you try the next command?

pmm-admin add external --group=processes  --listen-port=9256

Screenshot 2023-07-17 at 15.12.46
Screenshot 2023-07-17 at 15.12.461531Ă—684 33 KB

Screenshot 2023-07-17 at 15.13.09
Screenshot 2023-07-17 at 15.13.09960Ă—590 56.1 KB

1 Like
3

Well… I feel like a ID10T… Seems As I was scripting this from the time I started the service to the time I did the add external monitor was to quick. The service and port was not up which resulted in the failures… Once I put a sleep 5 between the enable/start service and the pmm-admin add things worked fine…
’

Of course when I was testing the 9256 the service had already started and I saw results so I “assumed”

What was throwing me off is if you attempt to add the external when it is not up it comes back with the error of needing the server url…

4

Ok one last item… I see the server shows it as unknown and marks it in Red. I do remember that as it is not part of pmm that the status is unknown… Is there a trick to change that somehow?