# PMM can not load Alertmanager config **URL:** \ **Category:** Percona Monitoring and Management (PMM)\ **Tags:** pmm\ **Created:** [September 14, 2024, 3:12am UTC](https://forums.percona.com/t/pmm-can-not-load-alertmanager-config/33135 "2024-09-14T03:12:27Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![Luke03011](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/luke03011/32/16409_2.png) [@Luke03011](https://forums.percona.com/u/Luke03011)\ **Post date:** [September 14, 2024, 3:12am UTC](https://forums.percona.com/t/pmm-can-not-load-alertmanager-config/33135/1 "2024-09-14T03:12:27Z")
## Description: PMM complains `Error loading Alertmanager config`. The error log in `grafana.log` ```auto logger=ngalert.multiorg.alertmanager t=2024-09-14T02:32:35.100569928Z level=error msg="failed to apply Alertmanager config for org" org=1 id=23 err="failed to parse Alertmanager config: unable to parse Alertmanager configuration: invalid character '{' after object key:value pair" ``` My Alertmanager config is like: ```auto # Managed by pmm-managed. DO NOT EDIT. --- global: resolve_timeout: 0s smtp_require_tls: false slack_api_url: https://hooks.slack.com/services/xxxx/xxxxx route: receiver: empty continue: false routes: - receiver: /channel_id/87c95895-16d7-4946-9c41-da3251d14452 match: rule_id: /rule_id/8e92f583-b05e-4334-be74-81e7fea84aff continue: false - receiver: /channel_id/87c95895-16d7-4946-9c41-da3251d14452 match: rule_id: /rule_id/b2ec7dfc-c2e3-4b10-a28f-bf212627b6d4 continue: false - receiver: /channel_id/87c95895-16d7-4946-9c41-da3251d14452 match: rule_id: /rule_id/5d6e752d-d387-48b6-9b8a-872baed1e534 continue: false - receiver: /channel_id/87c95895-16d7-4946-9c41-da3251d14452 match: rule_id: /rule_id/8a79828e-19fc-448d-a195-57bce5563023 continue: false - receiver: /channel_id/87c95895-16d7-4946-9c41-da3251d14452 match: rule_id: /rule_id/b2541f4b-ebaf-4f5f-986f-ae09ed2b97ff continue: false - receiver: /channel_id/87c95895-16d7-4946-9c41-da3251d14452 match: rule_id: /rule_id/542665f8-ce15-43a7-beb9-2151e8f683ae continue: false - receiver: /channel_id/87c95895-16d7-4946-9c41-da3251d14452 match: rule_id: /rule_id/e202c37b-27a7-4aec-9dd8-815bd353ccd5 continue: false - receiver: /channel_id/87c95895-16d7-4946-9c41-da3251d14452 match: rule_id: /rule_id/9b79417c-52b0-48c3-aa40-a1b6a719b44a continue: false - receiver: /channel_id/87c95895-16d7-4946-9c41-da3251d14452 match: rule_id: /rule_id/3beea76b-f932-4ca5-abd5-0b9f5a1e59c2 continue: false repeat_interval: 30m receivers: - name: empty - name: disabled - name: /channel_id/87c95895-16d7-4946-9c41-da3251d14452 slack_configs: - send_resolved: true channel: slack-ch2 title: '[{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}]' text: |- {{ range .Alerts -}} *Alert:* {{ if .Labels.severity }}`{{ .Labels.severity | toUpper }}`{{ end }} {{ .Annotations.summary }} *Description:* {{ .Annotations.description }} *Details:* {{ if .Labels.node_name }} • *node_name:* `{{ .Labels.node_name }}` {{ end }}{{ if .Labels.node_id }} • *node_id:* `{{ .Labels.node_id }}` {{ end }}{{ if .Labels.service_name }} • *service_name:* `{{ .Labels.service_name }}` {{ end }}{{ if .Labels.service_id }} • *service_id:* `{{ .Labels.service_id }}` {{ end }}{{ if .Labels.service_type }} • *service_type:* `{{ .Labels.service_type }}` {{ end }}{{ if .Labels.rule_id }} • *rule_id:* `{{ .Labels.rule_id }}` {{ end }}{{ if .Labels.alertgroup }} • *alertgroup:* `{{ .Labels.alertgroup }}` {{ end }}{{ if .Labels.template_name }} • *template_name:* `{{ .Labels.template_name }}` {{ end }}{{ if .Labels.severity }} • *severity:* `{{ .Labels.severity }}` {{ end }}{{ if .Labels.agent_id }} • *agent_id:* `{{ .Labels.agent_id }}` {{ end }}{{ if .Labels.agent_type }} • *agent_type:* `{{ .Labels.agent_type }}` {{ end }}{{ if .Labels.job }} • *job:* `{{ .Labels.job }}` {{ end }} {{ end }} short_fields: false link_names: false templates: [] ``` Error on the UI: ![Screenshot 2024-09-14 at 11.11.04](https://us1.discourse-cdn.com/flex019/uploads/percona1/original/3X/6/8/681601913f8d6769c9f2cc96e4b5d4596640b90b.png) ## Steps to Reproduce: I have no idea how to reproduce this, After upgrading from `2.39.0` to `2.42.0` for my production PMM, it may appear, and I just noticed it a few days ago, but strangely, my labs PMM works file after upgrading to `2.42.0`, so I do not know whether it’s upgrade related, but prod PMM does not send alerts since my upgrade. ## Version: PMM 2.42.0 ## Logs: The error log in `grafana.log` ```auto logger=ngalert.multiorg.alertmanager t=2024-09-14T02:32:35.100569928Z level=error msg="failed to apply Alertmanager config for org" org=1 id=23 err="failed to parse Alertmanager config: unable to parse Alertmanager configuration: invalid character '{' after object key:value pair" ``` No useful logs in `alertmanager.log` ## Expected Result: The steps to fix this. Or at least the ability to reconfigure the alertmanager. ---
**Author:** ![matthewb](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/matthewb/32/34_2.png) [@matthewb](https://forums.percona.com/u/matthewb)\ **Post date:** [September 16, 2024, 2:10pm UTC](https://forums.percona.com/t/pmm-can-not-load-alertmanager-config/33135/2 "2024-09-16T14:10:46Z")
Change your config to a simple message without the `{{ }}` template variables. Verify that this works and is parsed. Then, add one or two lines. Ensure parsed. Repeat until you find the issue with the errant { character. ---
**Author:** ![Luke03011](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/luke03011/32/16409_2.png) [@Luke03011](https://forums.percona.com/u/Luke03011)\ **Post date:** [September 18, 2024, 9:06am UTC](https://forums.percona.com/t/pmm-can-not-load-alertmanager-config/33135/3 "2024-09-18T09:06:09Z")
Thank you @matthewb I removed all {{ }} from the `/etc/alertmanager.yml`, and restart the `alertmanager` and even restart the pmm server docker, it still show the same error. I seems the `/etc/alertmanager.yml` is not relevant. And even I use a empty config: ```auto --- # You can edit this file; changes will be preserved. route: receiver: empty routes: [] repeat_interval: 1h receivers: - name: empty ``` The error is still there. ---
**Author:** ![matthewb](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/matthewb/32/34_2.png) [@matthewb](https://forums.percona.com/u/matthewb)\ **Post date:** [September 18, 2024, 3:24pm UTC](https://forums.percona.com/t/pmm-can-not-load-alertmanager-config/33135/4 "2024-09-18T15:24:54Z")
Hmm. I’m wondering if this config is stored using the internal pgsql database and it only reads the .yml file on first startup. You might need to start over a new PMM install to correct this, unfortunately. ---
**Author:** ![Luke03011](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/luke03011/32/16409_2.png) [@Luke03011](https://forums.percona.com/u/Luke03011)\ **Post date:** [September 19, 2024, 2:40am UTC](https://forums.percona.com/t/pmm-can-not-load-alertmanager-config/33135/5 "2024-09-19T02:40:04Z")
@matthewb is there a way to reset the alert configuration, e.g. reset it to the default setting when I deploy the PMM server. ---
**Author:** ![matthewb](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/matthewb/32/34_2.png) [@matthewb](https://forums.percona.com/u/matthewb)\ **Post date:** [September 19, 2024, 6:55pm UTC](https://forums.percona.com/t/pmm-can-not-load-alertmanager-config/33135/6 "2024-09-19T18:55:27Z")
I’m unsure if that is possible. The best option is to start a new PMM container and migrate over to it, unfortunately.