Hi
I am running PMM client on a cloud machine that comes up and down daily to save cost, but each time it comes up it has a different IP address
This means that some of the stats are not being recorded when it comes back up again if its based on IP
Reading some posts it sounds like restarting all of the services on restart is what you need to do but it seems like PMM2 doesnt have a restart command
ubuntu@ip-172-31-23-20:~$ sudo pmm-admin restart --all
pmm-admin: error: expected command but got "restart", try --help
Any advice on what I need to run on start of the server in order to fix this problem?
Thanks
To try and kick start it again I removed and added the service again
This has started the stats to be recorded back into the PMM server but its coming in under the same OLD IP address
Thats weird…
Hi Andrew,
pmm-admin is required pmm-agent service.
So this service can be restarted.
pmm-agent has to be reconfigured in order to communicate with a new pmm-server IP address.
Please run pmm-admin config with --server-url flag.
Hi @adivinho
thanks. No its not the pmm-server IP that is changing its the client IP that is changing so when I re-added the service by going delete and then add it seems to be recording stats again on the PMM server but still under the old client IP which seems weird.
If I restart the pmm-agent service will that sort out the change in Client IP?
What version of PMM do you use?
ubuntu@ip-172-31-23-20:~$ sudo pmm-admin --version
ProjectName: pmm-admin
Version: 2.10.1
PMMVersion: 2.10.1
Timestamp: 2020-09-22 11:35:37 (UTC)
FullCommit: 2d1a57bd14b4163cb1a5d99c2debb6afbe766ef8
VictoriaMetrics replaces Prometheus in the latest PMM version 2.12.0. VictoriaMetrics supports both PUSH (client to server) and PULL metrics collection modes. So if an exporter is working in the PUSH mode then metrics will be collected even if client’s ip had changed.
You may find some additional details here
https://www.percona.com/blog/2020/12/01/foiled-by-the-firewall-a-tale-of-transition-from-prometheus-to-victoriametrics/
Hi @adivinho
Thanks for the update. So are you saying there is no way around this using 2.10 and prometheus?
In order for me to get this to work I need to upgrade and start using VictoriaMetrics?
Am I still able to use the Prometheus Apache exporter or will that need to change as well?
Correct, prior to 2.12.0 the client registered itself with PMM Server and defined either an IP or Hostname that the pmm server used connect back to the client to PULL metrics from…so if the DB Server (pmm-client) was registered at 192.168.0.20 but after reboot was found at 192.168.0.21 PMM server would have no way of reestablishing communication…there ARE a few ways around in older versions the simplest being to register with the --host using a DNS name but you’ll need to make sure that you’re dynamically updating DNS when you pull a DHCP lease for that to work. You can also create a startup script to reregister the node on boot but it’s clunky and you’d have to both register the node (pmm-admin config) and each exporter (pmm-admin add).
The best solution is actually to upgrade to the 2.12.0 release (both client and server) and register your nodes with the --metrics-mode=push flag. With respect to your Apache Exporter, you’re using the “external exporter” feature of PMM which also supports push metrics (you’ll still have to reregister with the --metrics-mode=push).
Hi @steve.hoffman
Thanks for your tips
The DNS route would work for me as I am dynamically updating the DNS when the server comes back up again so that would certainly work.
So I guess the question then is how do I register an external service using DNS rather than IP?
Hmmm…I don’t have an external exporter installed handy to test with right now, but in theory you’d register the pmm client side using it’s dns name of “clientname.local”:
pmm-admin config --server-insecure-tls --server-url=https://<username>:<password>@pmm-server-hostanme clientname.local generic clientname
then when you go to add the external service (assume it’s running on port 5678)
pmm-admin add external --listen-port=5678
it will generate a URI on the pmm-server using the hostname for the NodeID you’re registering from that uses the hostname vs the IP. Just make sure the pmm server can resolve that hostname and you should be good even if the IP changes as long as server can resolve the new IP reasonably fast!
Hi @steve.hoffmanThanks for the tip. So its when I register the client rather than the service
I was using the following command
sudo pmm-admin config --server-insecure-tls --server-url=https://admin:<password>@<ip>
So I need to add the extra bits. If my dns was www.test.com would the command look as follows?
sudo pmm-admin config --server-insecure-tls --server-url=https://admin:<password>@<ip> www.test.com generic test.com
Not 100% sure what to put in those last 3 fields you suggested?
Sweet! Looks like @adivinhoproved it works the way we thought it would!!!
Here’s a breakdown of the last three positional arguments based on your example above:
www.test.com generic test.com
the first one (www.test.com) is either the IP or fully qualified DNS name of the system and whatever that resolves to MUST be reachable from the pmm-server
the second one (generic) is the node type, it can only be “generic” or “container”…if you don’t put this argument in it will assume “generic”
the final one (test.com) is the node name which we attempt to default to the local hostname but this allows you to override it to something easier to understand like “eCommerceDB” and makes display nicer in the UI.
You can see the full list by running ‘pmm-admin config --help’ on any client and you’ll also see the auto-detected defaults for the above.
Hi @steve.hoffman @adivinho
Thanks for your help on this
So I have tried what you said, I re-added the config using the 3 additional parameters and that is now done, but then I have re-added the external service and according to PMM admin the service and agents are up and running but I am not getting any stats into PMM server?
Any ideas on what I should look at as to why its not working? Makes me wonder if the change has broken something.
Hi Andrew,
You should check if external exporters are provided any metrics.
e.g.
curl http://localhost:5678
Also you can look at the results of going to "https:///prometheus/targets and you’ll see what the status is from the pmm servers perspective. it’s possible the pmm server cannot resolve the name (but if the container host can see it the container should too…). knowing what you see with this and @adivinho’s suggestion will help us see where to go next…
