I had an alert set up on MySQL Down, which sent me an email whenever a system went down. This has worked fine until fairly recently. I upgraded to 2.31 a week or so ago so I can only assume it’s related to this.
Today a system went down and I didn’t get an email.
I am unable to create new alerts - “Failed to save rule: Internal server error.; Internal server error.” on the webpage and:
logger=context traceID=00000000000000000000000000000000 userId=5 orgId=1 uname=psumner t=2022-11-02T13:29:27.62533991Z level=error msg="Error from access control system" error="could not resolve datasources:id:1: data source not found" accessErrorID=ACE2718563073 logger=context traceID=00000000000000000000000000000000 userId=5 orgId=1 uname=psumner t=2022-11-02T13:29:27.625439734Z level=info msg="Request Completed" method=GET path=/api/datasources/1 status=403 remote_addr=127.0.0.1 time_ms=2 duration=2.428558ms size=166 referer= traceID=00000000000000000000000000000000
This from grafana.log
Any clues where to start looking? I’ve tried various created and default templates, tried creating alerts in existing folder, and also created a new folder for the alerts. Same issue.
I’ve tried disabling and re-enabling Alerting in the settings menu and the problem persists.
Since writing this I’ve found the python script to migrate alerts. I’ve run it and…
[root@rsnpmm01 ~]# python3 alertrulemigration.py -uadmin -padmin Request existing IA rules. Request existing IA rules done. There are no rules to migrate, exiting.