# Prometheus alert manager failed to start **URL:** \ **Category:** PMM 2.x\ **Tags:** pmm, mysql\ **Created:** [February 5, 2021, 11:35pm UTC](https://forums.percona.com/t/prometheus-alert-manager-failed-to-start/9124 "2021-02-05T23:35:49Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![deepthip](https://avatars.discourse-cdn.com/v4/letter/d/dc4da7/32.png) [@deepthip](https://forums.percona.com/u/deepthip)\ **Post date:** [February 5, 2021, 11:35pm UTC](https://forums.percona.com/t/prometheus-alert-manager-failed-to-start/9124/1 "2021-02-05T23:35:50Z")
Hi I’m trying to integrate Prometheus alertmanger with PMM 2.14 and following this document someone shared on this forum > **[Integrating Prometheus Alert Manager with Percona Monitoring and Management...](https://playwithmysql.wordpress.com/2020/04/11/integrating-prometheus-alert-manager-with-percona-monitoring-and-management-pmm/)** > > What is Prometheus Alert Manager? Alert Manager is an open source tool used to route the alerts generated by Prometheus to your receiver integration like Slack, PagerDuty, VictorOps, Email, WeChat,… However, Its failing to start the alertmanager . I don’t see any error but It shows that it failed when I look at the status [root@xxxxx]# systemctl status alertmanager ● alertmanager.service - Alert Manager Loaded: loaded (/usr/lib/systemd/system/alertmanager.service; enabled; vendor preset: disabled) Active: failed (Result: start-limit) since Fri 2021-02-05 23:03:13 UTC; 12s ago Process: 17571 ExecStart=/usr/local/bin/alertmanager --config.file=/etc/alertmanager/alertmanager.yml --storage.path=/data/alertmanager (code=exited, status=1/FAILURE) Main PID: 17571 (code=exited, status=1/FAILURE) Feb 05 23:03:13 xxx.systemd[1]: Unit alertmanager.service entered failed state. Feb 05 23:03:13 xxx. systemd[1]: alertmanager.service failed. Feb 05 23:03:13 xxx. systemd[1]: alertmanager.service holdoff time over, scheduling restart. Feb 05 23:03:13 xxx.systemd[1]: Stopped Alert Manager. Feb 05 23:03:13 xxx systemd[1]: start request repeated too quickly for alertmanager.service Feb 05 23:03:13 xxx systemd[1]: Failed to start Alert Manager. Feb 05 23:03:13 xxx systemd[1]: Unit alertmanager.service entered failed state. Feb 05 23:03:13 xxx systemd[1]: alertmanager.service failed. alertmanager.yml file [root@xxx]# cat /etc/alertmanager/alertmanager.yml global: resolve\_timeout: 5m route: group\_by: [‘alertname’] group\_wait: 10s group\_interval: 10s repeat\_interval: 1h receiver: ‘web.hook’ receivers: - name: ‘web.hook’ webhook\_configs: - url: ‘[http://127.0.0.1:5001/](http://127.0.0.1:5001/)’ inhibit\_rules: - source\_match: severity: ‘critical’ target\_match: severity: ‘warning’ equal: [‘alertname’, ‘dev’, ‘instance’] Please suggest !! Any help is appreciated !! ---
**Author:** ![Vadim\_Yalovets](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/vadim_yalovets/32/43_2.png) [@Vadim\_Yalovets](https://forums.percona.com/u/Vadim_Yalovets)\ **Post date:** [February 6, 2021, 11:27am UTC](https://forums.percona.com/t/prometheus-alert-manager-failed-to-start/9124/2 "2021-02-06T11:27:54Z")
Hi Deepthip, You should verify your systemd unit file /etc/systemd/system/alertmanager.service Here is an example of a unit file on my installation: ![Screenshot_20210206_132736](https://us1.discourse-cdn.com/flex019/uploads/percona1/original/2X/4/4998a49c80be66f5779a2ea0b6147151c1a52cbe.png) ---
**Author:** ![Vadim\_Yalovets](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/vadim_yalovets/32/43_2.png) [@Vadim\_Yalovets](https://forums.percona.com/u/Vadim_Yalovets)\ **Post date:** [February 6, 2021, 11:34am UTC](https://forums.percona.com/t/prometheus-alert-manager-failed-to-start/9124/3 "2021-02-06T11:34:49Z")
Next command can be used for a validation. `systemd-analyze verify /etc/systemd/system/alertmanager.service` Also please notice that PMM 2.14.0 already has “Integrated Alerting” functionality. It’s disabled by default. ![Screenshot_20210206_133422](https://us1.discourse-cdn.com/flex019/uploads/percona1/original/2X/5/5474c51b74ac09237dd5389ae2c5b181e09a36ee.png) ---
**Author:** ![deepthip](https://avatars.discourse-cdn.com/v4/letter/d/dc4da7/32.png) [@deepthip](https://forums.percona.com/u/deepthip)\ **Post date:** [February 9, 2021, 10:49pm UTC](https://forums.percona.com/t/prometheus-alert-manager-failed-to-start/9124/4 "2021-02-09T22:49:48Z")
Thank adivinho, The issue was since I enable Integrated alerting , the port was already in use , so I had to use a different port. I was able to start it with a different port. Like you said , I started off using Intergrated Alerting but heard from the support that its not GA and still a technical preview and is not ready for production servers. Even otherwise, I’m struggling to make all the alerts work with Integrated Alerting. Like for eg. I get alerted when mysql is down but not for OS level alerts like CPU, Memory etc. I’m using the built in templates for now . Have you been using it ? if yes , how did it go for you ? Please suggest what I’m missing that I can’t get alerted for OS level alerts . ---
**Author:** ![Vadim\_Yalovets](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/vadim_yalovets/32/43_2.png) [@Vadim\_Yalovets](https://forums.percona.com/u/Vadim_Yalovets)\ **Post date:** [February 10, 2021, 10:27am UTC](https://forums.percona.com/t/prometheus-alert-manager-failed-to-start/9124/5 "2021-02-10T10:27:35Z")
There is [an issue](https://jira.percona.com/browse/PMM-7467) with alerts for symbol “less than”. You may check if your alert rules are affected by next command `docker exec -it /usr/bin/grep -r lt /etc/ia/rules/` e.g. ![Screenshot_20210210_122028](https://us1.discourse-cdn.com/flex019/uploads/percona1/original/2X/3/3f19cba30de5591ce78cbd939b0f305e201d93fc.png) So build-in template “Node out of memory” isn’t working due to the issue. But other node templates have to work. ---
**Author:** ![deepthip](https://avatars.discourse-cdn.com/v4/letter/d/dc4da7/32.png) [@deepthip](https://forums.percona.com/u/deepthip)\ **Post date:** [February 10, 2021, 4:12pm UTC](https://forums.percona.com/t/prometheus-alert-manager-failed-to-start/9124/6 "2021-02-10T16:12:53Z")
Understood . One more thing I forgot to mention above is I’m not using a docker. I’m using the AWS AMI provided from the market place and I’m monitoring Aurora MySQL clusters. So it has to get the OS related information via the cloudmetrics . I checked and I think enhanced monitoring is turned on for the RDS instances . Anything else I could be missing here ? Is Integrated alerting even supposed to be working for AWS RDS Instances being monitored by AWS AMI from market place ? So far I do not have any custom alerts . I’m still working around with Built in alerts ---
**Author:** ![Vadim\_Yalovets](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/vadim_yalovets/32/43_2.png) [@Vadim\_Yalovets](https://forums.percona.com/u/Vadim_Yalovets)\ **Post date:** [February 12, 2021, 11:28am UTC](https://forums.percona.com/t/prometheus-alert-manager-failed-to-start/9124/7 "2021-02-12T11:28:53Z")
You may check collected RDS metrics in Explore. `count by ( __name__ ) ({ __name__ =~"rdsosmetrics_.*", node_name="rds-mysql57"})` ![Screenshot_20210212_132751](https://us1.discourse-cdn.com/flex019/uploads/percona1/original/2X/7/7dcb9d9eff166bb8b6056fd15161d4eab1013368.png) We have got an RDS instance that is monitored by our PMM demo instance. Here is [a link](https://pmmdemo.percona.com/graph/explore?orgId=1&left=%5B%22now-1h%22,%22now%22,%22Metrics%22,%7B%22expr%22:%22count%20by%20( __name__ )%20(%7B __name__ %3D~%5C%22rdsosmetrics_.*%5C%22,%20node_name%3D%5C%22rds-mysql57%5C%22%7D)%22,%22hide%22:false%7D,%7B%22ui%22:%5Bfalse,true,true,%22none%22%5D%7D%5D) to the site. ---
**Author:** ![deepthip](https://avatars.discourse-cdn.com/v4/letter/d/dc4da7/32.png) [@deepthip](https://forums.percona.com/u/deepthip)\ **Post date:** [February 15, 2021, 6:11am UTC](https://forums.percona.com/t/prometheus-alert-manager-failed-to-start/9124/8 "2021-02-15T06:11:29Z")
Got it ! Is this normal for the RDS hosts that are being monitored to have some graphs as no data returned . Please note that this node is being monitored and returned results from you query above . ![image](https://us1.discourse-cdn.com/flex019/uploads/percona1/original/2X/f/f9a5cc39ef2a42b52ee2cbf94b14047a2d92f06b.png) ---
**Author:** ![deepthip](https://avatars.discourse-cdn.com/v4/letter/d/dc4da7/32.png) [@deepthip](https://forums.percona.com/u/deepthip)\ **Post date:** [February 15, 2021, 6:14am UTC](https://forums.percona.com/t/prometheus-alert-manager-failed-to-start/9124/9 "2021-02-15T06:14:13Z")
The reason I ask is , I still can’t get the CPU alerts no matter how low I set the threshold. So . just wondering if it has to do anything with the “CPU saturation and max core usage graph” ---
**Author:** ![Vadim\_Yalovets](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/vadim_yalovets/32/43_2.png) [@Vadim\_Yalovets](https://forums.percona.com/u/Vadim_Yalovets)\ **Post date:** [February 15, 2021, 9:02am UTC](https://forums.percona.com/t/prometheus-alert-manager-failed-to-start/9124/10 "2021-02-15T09:02:59Z")
You may use the attached alert template that is based on RDS metric _node\_cpu\_average_. It can be added on the tab “Alert Rule Templates” and used for creating alert rule on the tab “Alert Rules”. ![Screenshot_20210215_105937](https://us1.discourse-cdn.com/flex019/uploads/percona1/original/2X/e/e89b67c26ff90a791b8a4e3da6b63a57b3ab6f7e.png) [rds\_cpu.txt](https://forums.percona.com/uploads/short-url/wD8vloqbCZClHVmY27XaySf3GXs.txt) (691 Bytes) ---
**Author:** ![deepthip](https://avatars.discourse-cdn.com/v4/letter/d/dc4da7/32.png) [@deepthip](https://forums.percona.com/u/deepthip)\ **Post date:** [February 16, 2021, 1:12am UTC](https://forums.percona.com/t/prometheus-alert-manager-failed-to-start/9124/11 "2021-02-16T01:12:17Z")
Awesome !! This helped ! I was finally able to get the CPU alerts . Thanks much adivinho !! On the same note ,can you suggest on the below ? 1. So am I correct to assume that some of the in-built alert templates may not work for RDS mysql environments ? 2. Can you please share any other alert templates for RDS, such as Out of memory, replication for MySQL etc . 3. Can you point me to any sources explaining the RDS metrics that might be useful in building the alerts 4. Finally how can I make these alerts appear in a more human readable format ![image](https://us1.discourse-cdn.com/flex019/uploads/percona1/original/2X/0/07687cba6d7e005092ac968ff62e8214f7f464b9.png) Thanks again !! ---
**Author:** ![Vadim\_Yalovets](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/vadim_yalovets/32/43_2.png) [@Vadim\_Yalovets](https://forums.percona.com/u/Vadim_Yalovets)\ **Post date:** [February 17, 2021, 2:36pm UTC](https://forums.percona.com/t/prometheus-alert-manager-failed-to-start/9124/12 "2021-02-17T14:36:08Z")
> [@deepthip](#): > > - So am I correct to assume that some of the in-built alert templates may not work for RDS mysql environments ? Mainly all metrics names are equal for rds exporter and node, mysqld exporters. But there is an exception for CPU usage. I will create a ticket for making build-in template “Node high CPU load” compatible with RDS metrics. Other templates use compatible expressions. > [@deepthip](#): > > - Can you please share any other alert templates for RDS, such as Out of memory, replication for MySQL etc . We have to wait the next release when [the issue](https://jira.percona.com/browse/PMM-7467) is solved. It’s pretty hard to invent an alert without using symbol “less then”. > [@deepthip](#): > > - Can you point me to any sources explaining the RDS metrics that might be useful in building the alerts You may find metric names and descriptions in next test data files for rds\_exporter. > > > //github.com/percona/rds\_exporter/tree/main/enhanced/testdata > [@deepthip](#): > > - Finally how can I make these alerts appear in a more human readable format It will be changed in future releases. ---
**Author:** ![deepthip](https://avatars.discourse-cdn.com/v4/letter/d/dc4da7/32.png) [@deepthip](https://forums.percona.com/u/deepthip)\ **Post date:** [February 23, 2021, 5:37am UTC](https://forums.percona.com/t/prometheus-alert-manager-failed-to-start/9124/13 "2021-02-23T05:37:20Z")
Understood ! Thanks adivinho