Alerting using Custom template - Alerting but with Extraneous Labels


I am creating an alert to tell me if an instance in my MySQL Cluster has a Group Replication Node OFFLINE. I create the rule and it fires fine and is alerting. However, it is not getting pushed to the Notification Policy to alert me via Slack because the Labels the alert generates are more than what I have set for the Notification Policy. I don’t know why that is happening. It seems the labels it generates matches the columns returned from the Preview Alert button while editing the rule. I don’t know how to control that.

Steps to Reproduce:

Notification Policy Set to alert on 2 labels:severity=critical and grafana_folder=MySQL Non-Production.

For the rule, the Metric expression used is mysql_perf_schema_replication_group_member_info{service_name="ia-se-vudb3-123-mysql", member_state="OFFLINE"}

The rule is being stored in Folder MySQL Non-Production.
The Group is default-alert-group.
The notifications has two labels for it: severity=critical and grafana_folder=MySQL Non-Production.

Also when viewing the rule, the Labels display several additional labels which were automatically created: alert name, datasource_uid, red_id. This may have a factor on why there are so many Labels when the Alert is firing and Alerting. I’ll provide screen shot.



Any help would be appreciated.

Well… after bashing my head on this yesterday trying to figure out why this generated alert wasn’t being pushed to a Notification Policy. It magically started working this morning! Not quite sure what changed if anything. I retested several times just to make sure. Phew.

1 Like

Happy to know it’s worked for you, @devdewboy.
The Alert template looks interesting. Can you share the full JSON so others can reuse it?