PMM2 Alerting not working since upgrade to 2.31

I had an alert set up on MySQL Down, which sent me an email whenever a system went down. This has worked fine until fairly recently. I upgraded to 2.31 a week or so ago so I can only assume it’s related to this.

Today a system went down and I didn’t get an email.

I am unable to create new alerts - “Failed to save rule: Internal server error.; Internal server error.” on the webpage and:

logger=context traceID=00000000000000000000000000000000 userId=5 orgId=1 uname=psumner t=2022-11-02T13:29:27.62533991Z level=error msg="Error from access control system" error="could not resolve datasources:id:1: data source not found" accessErrorID=ACE2718563073

logger=context traceID=00000000000000000000000000000000 userId=5 orgId=1 uname=psumner t=2022-11-02T13:29:27.625439734Z level=info msg="Request Completed" method=GET path=/api/datasources/1 status=403 remote_addr=127.0.0.1 time_ms=2 duration=2.428558ms size=166 referer= traceID=00000000000000000000000000000000

This from grafana.log

Any clues where to start looking? I’ve tried various created and default templates, tried creating alerts in existing folder, and also created a new folder for the alerts. Same issue.

I’ve tried disabling and re-enabling Alerting in the settings menu and the problem persists.

[Edit]

Since writing this I’ve found the python script to migrate alerts. I’ve run it and…

[root@rsnpmm01 ~]# python3 alertrulemigration.py -uadmin -padmin
Request existing IA rules.
Request existing IA rules done.
There are no rules to migrate, exiting.
1 Like

hi where can i find this python file . it is in the server please give to me .py file location or where can i find thank you.

1 Like
1 Like

thank you for your answer but when i run this command in server ;
python ia_migration.py -u admin -p admin

I get an error in bellow

(python3: can’t open file ‘ia_migration.py’: [Errno 2] No such file or directory)

1 Like

@k3r1m you may need to do it like python3 /path/to/ia_migration.py -u admin - p admin

@psumner I’m posting this internally to see if the team that works on alerting can offer some help.

1 Like

Hello psumner,It might be helpful to know - you upgraded from what previous version to 2.31.0?

2 Likes

that’s the issue I do not know the path where is migration.py what can i find it ?

1 Like

ahh, you have to download it here:

and then run the command from the folder you downloaded the file to.

1 Like

Oh I thought I’d included that. Previous version was 2.30

1 Like