Integrated Alerts - Is there no resolution message for when an alert returns to normal?

Howdy,

I’ve just gotten into setting up the new alert system. I’m testing with the “mysql down” alert rule.

So far, it works! I can get an alert into a Slack channel when I run tests by taking down the mysql service on a node.

Is there a way to receive an “alert resolved”-type of message in Slack when the service returns to normal? Is it an extra config on my end, or a new feature that will come along eventually? Not possible/planned?

Thanks,

1 Like

Hello. For now, there is no way for you to get a Slack notification if the alert was resolved. It can be requested as a new feature.
So far we have the ability to notify about resolved alerts for the Webhook notification channel.

1 Like

Thank you, @vasyl_yurkovych.

I’ll look to see if it isn’t already a feature request (honestly, how could it not be?) and enter one.

1 Like

@bccmu looks like it is implemented via API. You could send a POST request ‘http://{PMM_SERVER_IP}/v1/management/ia/Channels/Add’

with body like:
‘{“summary”:“Slack NC”,“slack_config”:{“channel”:“some_channel_name”, “send_resolved”: true}}’

This should enable notifications for resolved alerts for this channel

1 Like

@bccmu Sorry for the confusion. But looks like we have an issue with the send_resolved flag. I’ve logged a bug for it.

1 Like

Ah, that’s a bummer! Thanks for logging the bug.

While I’ve got you here, can you tell me where on the PMM server’s filesystem I can find the Integrated Alerting built-in template files? I wanted to figure out how they work so I can create my own. I’ve been searching but haven’t tracked them down yet. Or, if better, point me at a “canonical” example of template to start from?

Thanks,

1 Like

@vasyl_yurkovych Sorry, forgot to @ you for this :point_up: Send me somewhere else if this is not your cup of tea.

1 Like

@bccmu you’re welcome! You can find them in our repo. Enjoy :slightly_smiling_face:

1 Like

@vasyl_yurkovych For what it’s worth, this did work for me. I get resolved messages in Slack now :+1:

2 Likes

Although it is a bit spotty, in that if two nodes fire an alert I might get a resolved notice for just one of them. A work in progress, I guess :slight_smile:

1 Like