PMM can not load Alertmanager config

Percona Monitoring and Management (PMM)
1

Description:

PMM complains Error loading Alertmanager config.

The error log in grafana.log

logger=ngalert.multiorg.alertmanager t=2024-09-14T02:32:35.100569928Z level=error msg="failed to apply Alertmanager config for org" org=1 id=23 err="failed to parse Alertmanager config: unable to parse Alertmanager configuration: invalid character '{' after object key:value pair"

My Alertmanager config is like:

# Managed by pmm-managed. DO NOT EDIT.
---
global:
    resolve_timeout: 0s
    smtp_require_tls: false
    slack_api_url: https://hooks.slack.com/services/xxxx/xxxxx
route:
    receiver: empty
    continue: false
    routes:
        - receiver: /channel_id/87c95895-16d7-4946-9c41-da3251d14452
          match:
            rule_id: /rule_id/8e92f583-b05e-4334-be74-81e7fea84aff
          continue: false
        - receiver: /channel_id/87c95895-16d7-4946-9c41-da3251d14452
          match:
            rule_id: /rule_id/b2ec7dfc-c2e3-4b10-a28f-bf212627b6d4
          continue: false
        - receiver: /channel_id/87c95895-16d7-4946-9c41-da3251d14452
          match:
            rule_id: /rule_id/5d6e752d-d387-48b6-9b8a-872baed1e534
          continue: false
        - receiver: /channel_id/87c95895-16d7-4946-9c41-da3251d14452
          match:
            rule_id: /rule_id/8a79828e-19fc-448d-a195-57bce5563023
          continue: false
        - receiver: /channel_id/87c95895-16d7-4946-9c41-da3251d14452
          match:
            rule_id: /rule_id/b2541f4b-ebaf-4f5f-986f-ae09ed2b97ff
          continue: false
        - receiver: /channel_id/87c95895-16d7-4946-9c41-da3251d14452
          match:
            rule_id: /rule_id/542665f8-ce15-43a7-beb9-2151e8f683ae
          continue: false
        - receiver: /channel_id/87c95895-16d7-4946-9c41-da3251d14452
          match:
            rule_id: /rule_id/e202c37b-27a7-4aec-9dd8-815bd353ccd5
          continue: false
        - receiver: /channel_id/87c95895-16d7-4946-9c41-da3251d14452
          match:
            rule_id: /rule_id/9b79417c-52b0-48c3-aa40-a1b6a719b44a
          continue: false
        - receiver: /channel_id/87c95895-16d7-4946-9c41-da3251d14452
          match:
            rule_id: /rule_id/3beea76b-f932-4ca5-abd5-0b9f5a1e59c2
          continue: false
    repeat_interval: 30m
receivers:
    - name: empty
    - name: disabled
    - name: /channel_id/87c95895-16d7-4946-9c41-da3251d14452
      slack_configs:
        - send_resolved: true
          channel: slack-ch2
          title: '[{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}]'
          text: |-
            {{ range .Alerts -}}
            *Alert:* {{ if .Labels.severity }}`{{ .Labels.severity | toUpper }}`{{ end }} {{ .Annotations.summary }}
            *Description:* {{ .Annotations.description }}
            *Details:*
            {{ if .Labels.node_name }}     • *node_name:* `{{ .Labels.node_name }}`
            {{ end }}{{ if .Labels.node_id }}     • *node_id:* `{{ .Labels.node_id }}`
            {{ end }}{{ if .Labels.service_name }}     • *service_name:* `{{ .Labels.service_name }}`
            {{ end }}{{ if .Labels.service_id }}     • *service_id:* `{{ .Labels.service_id }}`
            {{ end }}{{ if .Labels.service_type }}     • *service_type:* `{{ .Labels.service_type }}`
            {{ end }}{{ if .Labels.rule_id }}     • *rule_id:* `{{ .Labels.rule_id }}`
            {{ end }}{{ if .Labels.alertgroup }}     • *alertgroup:* `{{ .Labels.alertgroup }}`
            {{ end }}{{ if .Labels.template_name }}     • *template_name:* `{{ .Labels.template_name }}`
            {{ end }}{{ if .Labels.severity }}     • *severity:* `{{ .Labels.severity }}`
            {{ end }}{{ if .Labels.agent_id }}     • *agent_id:* `{{ .Labels.agent_id }}`
            {{ end }}{{ if .Labels.agent_type }}     • *agent_type:* `{{ .Labels.agent_type }}`
            {{ end }}{{ if .Labels.job }}     • *job:* `{{ .Labels.job }}`
            {{ end }}

            {{ end }}
          short_fields: false
          link_names: false
templates: []

Error on the UI:

Screenshot 2024-09-14 at 11.11.04
Screenshot 2024-09-14 at 11.11.041449×711 30.6 KB

Steps to Reproduce:

I have no idea how to reproduce this, After upgrading from 2.39.0 to 2.42.0 for my production PMM, it may appear, and I just noticed it a few days ago, but strangely, my labs PMM works file after upgrading to 2.42.0, so I do not know whether it’s upgrade related, but prod PMM does not send alerts since my upgrade.

Version:

PMM 2.42.0

Logs:

The error log in grafana.log

logger=ngalert.multiorg.alertmanager t=2024-09-14T02:32:35.100569928Z level=error msg="failed to apply Alertmanager config for org" org=1 id=23 err="failed to parse Alertmanager config: unable to parse Alertmanager configuration: invalid character '{' after object key:value pair"

No useful logs in alertmanager.log

Expected Result:

The steps to fix this. Or at least the ability to reconfigure the alertmanager.

2

Change your config to a simple message without the {{ }} template variables. Verify that this works and is parsed. Then, add one or two lines. Ensure parsed. Repeat until you find the issue with the errant { character.

3

Thank you @matthewb
I removed all {{ }} from the /etc/alertmanager.yml, and restart the alertmanager and even restart the pmm server docker, it still show the same error. I seems the /etc/alertmanager.yml is not relevant.

And even I use a empty config:

---
# You can edit this file; changes will be preserved.

route:
    receiver: empty
    routes: []
    repeat_interval: 1h
receivers:
    - name: empty

The error is still there.

4

Hmm. I’m wondering if this config is stored using the internal pgsql database and it only reads the .yml file on first startup. You might need to start over a new PMM install to correct this, unfortunately.

5

@matthewb is there a way to reset the alert configuration, e.g. reset it to the default setting when I deploy the PMM server.

6

I’m unsure if that is possible. The best option is to start a new PMM container and migrate over to it, unfortunately.