How exactly are you supposed to use the built in alertmanager.yml? I see there’s one in the pmm-server container at /etc/alertmanager.yml, I’m pretty sure that a previous build I used, I could copy my custom .yml file over the top of this and it would work (IE, it would show up as the config in http://localhost:9093/alertmanager/#/status) Now it seems like you have have a file named alertmanager.base.yml in /srv/alertmanager, but this doesn’t seem to pick up changes on the fly, so then I have to restart the container (instead of just having it built-in when i build the docker container) Also, it seems to be getting a default config from somewhere that has all of the default alerting options set, where is this config located in the container?
At this point PMM only supports an external integration with Alertmanager (Alertmanager and PMM Server being two different installs). As you have discovered we are working on an internal integration but it’s not officially released (and as such I highly recommend proceeding with caution) as many things are still changing about it’s implementation as we prepare to stabilize it for both PMM’s use as well as for users to incorporate their own needs. The only thing we’re using it for at the moment is for generating alerts for our Security Threat Tool and I believe the “configuration file” for that is actually dynamically generated on the fly and the plan is that user contributions will be merged in after being validated but we have to protect the intended use in the application space from unintentional errors in the user space. I’m guessing you stumbled upon it early in the inclusion process where we included it with no real usage to stabilize the integration and then added security threat tool on top of it. Unfortunately I can’t give a timeline for when it will officially released but what I would suggest is potentially using the integration feature under PMM Settings and pulling down an AlertManager container in parallel with your PMM server container until it’s “ready for prime time”.
Thank you for the clarification! About a month ago I had started off by spinning up my own alertmanager and messing around with it until i noticed that there was one that was a different one running already that seemed to be built-in, so, I thought that was the way to go!
Is there any way then to disable the included alertmanager? I guess I didn’t noticed that it caused conflicts in the past, but better safe than sorry?
/srv/alertmanager/alertmanager.base.yml is the single source of Alertmanager configuration. That will change in the future, but
alertmanager.base.yml content will be preserved and honored across upgrades and restarts. You can see that it is the only file currently used in
/etc/supervisord.d/alertmanager.ini. Please note that changes in this file are not preserved. Comments in both files say just so.
For Alertmanager to pick up a new configuration, you can restart it with
supervisorctl restart alertmanager. You can also stop it with
supervisorctl stop alertmanager and prevent it from being restarted with
supervisorctl remove alertmanager. But the last operation is highly not recommended. If Alertmanager is doing nothing, it does not consume a lot of resources, and removing it may break future upgrades.
Thanks for the info! I’ll disable the baked in one if I run into issues. I have since spun up my own alertmanager.
alexey.palazhchenko said: But the last operation is highly not recommended. If Alertmanager is doing nothing, it does not consume a lot of resources, and removing it may break future upgrades.Sorry for the late reply, this is actually causing me trouble. I'm trying to spin up docker images for PMM and the community supported Alertmanager on the same server, and they seem to be trying to bind the same ports or endpoints so sent alerts don't work correctly.