Integrated Alerts - Is there no resolution message for when an alert returns to normal?

Percona Monitoring and Management (PMM) PMM 2.x
#1

Howdy,

I’ve just gotten into setting up the new alert system. I’m testing with the “mysql down” alert rule.

So far, it works! I can get an alert into a Slack channel when I run tests by taking down the mysql service on a node.

Is there a way to receive an “alert resolved”-type of message in Slack when the service returns to normal? Is it an extra config on my end, or a new feature that will come along eventually? Not possible/planned?

Thanks,

1 Like
#2

Hello. For now, there is no way for you to get a Slack notification if the alert was resolved. It can be requested as a new feature.
So far we have the ability to notify about resolved alerts for the Webhook notification channel.

1 Like
#3

Thank you, @vasyl_yurkovych.

I’ll look to see if it isn’t already a feature request (honestly, how could it not be?) and enter one.

1 Like
#4

@bccmu looks like it is implemented via API. You could send a POST request ‘http://{PMM_SERVER_IP}/v1/management/ia/Channels/Add’

with body like:
‘{“summary”:“Slack NC”,“slack_config”:{“channel”:“some_channel_name”, “send_resolved”: true}}’

This should enable notifications for resolved alerts for this channel

1 Like
#5

@bccmu Sorry for the confusion. But looks like we have an issue with the send_resolved flag. I’ve logged a bug for it.

1 Like
#6

Ah, that’s a bummer! Thanks for logging the bug.

While I’ve got you here, can you tell me where on the PMM server’s filesystem I can find the Integrated Alerting built-in template files? I wanted to figure out how they work so I can create my own. I’ve been searching but haven’t tracked them down yet. Or, if better, point me at a “canonical” example of template to start from?

Thanks,

1 Like
#7

@vasyl_yurkovych Sorry, forgot to @ you for this :point_up: Send me somewhere else if this is not your cup of tea.

1 Like
#8

@bccmu you’re welcome! You can find them in our repo. Enjoy :slightly_smiling_face:

1 Like
#9

@vasyl_yurkovych For what it’s worth, this did work for me. I get resolved messages in Slack now :+1:

2 Likes
#10

Although it is a bit spotty, in that if two nodes fire an alert I might get a resolved notice for just one of them. A work in progress, I guess :slight_smile:

1 Like