How to customize slack alert message?

Percona Monitoring and Management (PMM)
1

I find the slack alert configuration file is /etc/alertmanager.yml and the heading line saying

Managed by pmm-managed. DO NOT EDIT

So editing this file directly will not work.

And the alert message template is defined in this file as:

receivers:
    - name: empty
    - name: disabled
    - name: /channel_id/b508fbb6-3479-42ff-b6de-8429a199f36d
      slack_configs:
        - send_resolved: true
          channel: slack-ch2
          title: '[{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}]'
          text: |-
            {{ range .Alerts -}}
            *Alert:* {{ if .Labels.severity }}`{{ .Labels.severity | toUpper }}`{{ end }} {{ .Annotations.summary }}
            *Description:* {{ .Annotations.description }}
            *Details:*
            {{ if .Labels.node_name }}     • *node_name:* `{{ .Labels.node_name }}`
            {{ end }}{{ if .Labels.node_id }}     • *node_id:* `{{ .Labels.node_id }}`
            {{ end }}{{ if .Labels.service_name }}     • *service_name:* `{{ .Labels.service_name }}`
            {{ end }}{{ if .Labels.service_id }}     • *service_id:* `{{ .Labels.service_id }}`
            {{ end }}{{ if .Labels.service_type }}     • *service_type:* `{{ .Labels.service_type }}`
            {{ end }}{{ if .Labels.rule_id }}     • *rule_id:* `{{ .Labels.rule_id }}`
            {{ end }}{{ if .Labels.alertgroup }}     • *alertgroup:* `{{ .Labels.alertgroup }}`
            {{ end }}{{ if .Labels.template_name }}     • *template_name:* `{{ .Labels.template_name }}`
            {{ end }}{{ if .Labels.severity }}     • *severity:* `{{ .Labels.severity }}`
            {{ end }}{{ if .Labels.agent_id }}     • *agent_id:* `{{ .Labels.agent_id }}`
            {{ end }}{{ if .Labels.agent_type }}     • *agent_type:* `{{ .Labels.agent_type }}`
            {{ end }}{{ if .Labels.job }}     • *job:* `{{ .Labels.job }}`
            {{ end }}

            {{ end }}

How can I customize the “text” field of the slack alert message? Thanks.

2 Likes
2

I find the alert message is hardcoded in the code services/alertmanager/alertmanager.go::formatSlackText:

func formatSlackText(labels ...string) string {
	const listEntryFormat = "{{ if .Labels.%[1]s }}     • *%[1]s:* `{{ .Labels.%[1]s }}`\n{{ end }}"

	text := "{{ range .Alerts -}}\n" +
		"*Alert:* {{ if .Labels.severity }}`{{ .Labels.severity | toUpper }}`{{ end }} {{ .Annotations.summary }}\n" +
		"*Description:* {{ .Annotations.description }}\n" +
		"*Details:*\n"
	for _, l := range labels {
		text += fmt.Sprintf(listEntryFormat, l)
	}

	text += "\n\n{{ end }}"

	return text
}

I can modify this file but I do not know how to rebuild the pmm-managed binary, could someone help me with this or point me to a document that may help? Thanks, I pull the code from GitHub and load it to GoLang IDE, but make failed, as I am a newbie to Go, it’s really complicated for me to build the pmm-managed binary without instructions.

2 Likes
3

I find a workaround. I can modify the /etc/alertmanager.yml and restart the alertmanager. But the modification will be lost if the docker is restarted.

2 Likes