How collect metrics from docker 9323/metrics to pmm ?
Hello @Yevhen,
You need to either publish the proper ports on the container so they are reachable for scraping, or make sure the PMM agent inside the container is in ‘push’ mode.
metrics are reachable from inside docker container ppm server
i tried to add it through gui
Inventory / Add service / Add external service
parse from url string
http://dns:9323/metrics
but it shows fail in result in status
in the same time, curl http://dns:9323/metrics works inside container
@Yevhen,
I am confused. Are you running MySQL inside a docker container? If so, you should ‘add external mysql service’ and point to the container’s mysql port of 3306.
No, docker now has prometheus compatible metrics exposed on port 9323 and I believe @Yevhen is trying to import those metrics into PMM to create graphs and dashboards about dockers performance.
Here’s a guide on getting external metrics (i.e. metrics not delivered by PMM’s built-in exporters for node, mysql, postgresql and mongo). You’ll likely want to have pmm-client running on the host with docker and register that client to pmm-server. You can then add external monitoring of docker from the command line on your docker host as outlined in the docs.
You could use the external-serverless approach, where PMM servers built-in client can reach out and scrape the metrics port but I don’t believe that metrics port is exposed outside the host by default (it’s bound to loopback/127.0.0.1 by default) so you’d have to open that port up on the external interface which may be more exposure than you intend…and I’m not even sure how to do that.
Once you get the external exporter configured metrics will start flowing to PMM and you can go to grafana’s site to see if there are any default dashboards you can import to PMM or you can create your own from scratch.
Yeh
I saw that guide
It is not clear for me due to i am new with prometheous
So i need external exporter for docker ?
I saw 2 docker exporters hub and cloud
Bouth deprecated in github
I thought prometheus can collect docker metrics by own fron that http
And yes i shered metrics at this moment with 0.0.0.0 in order to be reachable from anywhete for testing
Docker swarm + innodb cluster + pmm
I have pmm client on each node and i have metrics from nodes and mysql but i wand add docker metrics as well
so if you have pmm-client on the node you want to read metrics from you should be able to run something to the effect of:
pmm-admin add external --service-name=docker --listen-port=9323 --metrics-path=/metrics and that will register the service with pmm-server. From there the client will poll based on your default resolution settings and send the metrics directly to PMM server.
if you have metrics exposed on 0.0.0.0:9323 you should be able to first do a test from the pmm-server’s command line with curl or wget to verify pmm-server can actually scrape the metrics and you could use the GUI to configure PMM’s built-in client to scrape your docker metrics port.
pmm-admin add external --service-name=docker --listen-port=9323 --metrics-path=/metrics
Isn’t it synonyme to add service in inventory through gui ?
If It is, so as i said in the second messange
I do curl inside docker container pmm server successfully and i have fail state after adding external service with help of gui
The gui is actually doing external-serverless…but if you can curl to get the list of metrics then perhaps the issue isn’t a connectivity/network issue but a parsing issue.
Take a look in /srv/logs/pmm-managed.log on the PMM server (tail it while you’re trying to add the external exporter) and see if that sheds any light as to why the failure.
some suspicious log
time="2024-03-01T21:32:26.188+00:00" level=warning msg="VictoriaMetrics scrape configuration contains unsupported params: 2024/03/01 21:32:26 unprocessed command-line args left: [true]; the most likely reason is missing `=` between boolean flag name and value; see https://pkg.go.dev/flag#hdr-Command_line_flag_syntax\n" component=victoriametrics
time="2024-03-01T21:32:26.189+00:00" level=info msg="Configuration reloaded." component=victoriametrics
Hi @Yevhen, that warning isn’t related to your case and will be fixed in next patch release.
pmm-admin add external --service-name=docker --listen-port=9323 --metrics-path=/metrics
isn’t synonym for GUI one because in this case metrics will be collected by the local vmagent, not by the one installed on PMM.