Integrated Alerts - Is there no resolution message for when an alert returns to normal?

[Brian_Clark]

Howdy,

I’ve just gotten into setting up the new alert system. I’m testing with the “mysql down” alert rule.

So far, it works! I can get an alert into a Slack channel when I run tests by taking down the mysql service on a node.

Is there a way to receive an “alert resolved”-type of message in Slack when the service returns to normal? Is it an extra config on my end, or a new feature that will come along eventually? Not possible/planned?

Thanks,

[vasyl.yurkovych]

Hello. For now, there is no way for you to get a Slack notification if the alert was resolved. It can be requested as a new feature.
So far we have the ability to notify about resolved alerts for the Webhook notification channel.

[Brian_Clark]

Thank you, @vasyl.yurkovych.

I’ll look to see if it isn’t already a feature request (honestly, how could it not be?) and enter one.

[vasyl.yurkovych]

@Brian_Clark looks like it is implemented via API. You could send a POST request ‘http://{PMM_SERVER_IP}/v1/management/ia/Channels/Add’

with body like:
‘{“summary”:“Slack NC”,“slack_config”:{“channel”:“some_channel_name”, “send_resolved”: true}}’

This should enable notifications for resolved alerts for this channel

[vasyl.yurkovych]

@Brian_Clark Sorry for the confusion. But looks like we have an issue with the send_resolved flag. I’ve logged a bug for it.

[Brian_Clark]

Ah, that’s a bummer! Thanks for logging the bug.

While I’ve got you here, can you tell me where on the PMM server’s filesystem I can find the Integrated Alerting built-in template files? I wanted to figure out how they work so I can create my own. I’ve been searching but haven’t tracked them down yet. Or, if better, point me at a “canonical” example of template to start from?

Thanks,

[Brian_Clark]

@vasyl.yurkovych Sorry, forgot to @ you for this Send me somewhere else if this is not your cup of tea.

[vasyl.yurkovych]

@Brian_Clark you’re welcome! You can find them in our repo. Enjoy

[Brian_Clark]

@vasyl.yurkovych For what it’s worth, this did work for me. I get resolved messages in Slack now

[Brian_Clark]

Although it is a bit spotty, in that if two nodes fire an alert I might get a resolved notice for just one of them. A work in progress, I guess

[jhon.gomez]

Hello, I have the same need (send resolved alerts through the slack channel), however, when I send the POST request, it is not sent. Is it always the same url path? I have PMM version (2.27.0 (April 12, 2022))

[vasyl.yurkovych]

Hello @jhon.gomez ! Yes the endpoint should be the same? could you please share the url? and a response message and code?

[jhon.gomez]