# PMM2 Alerting not working since upgrade to 2.31 **URL:** \ **Category:** PMM 2.x\ **Created:** [November 2, 2022, 1:36pm UTC](https://forums.percona.com/t/pmm2-alerting-not-working-since-upgrade-to-2-31/18316 "2022-11-02T13:36:50Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![psumner](https://avatars.discourse-cdn.com/v4/letter/p/e8c25b/32.png) [@psumner](https://forums.percona.com/u/psumner)\ **Post date:** [November 2, 2022, 1:36pm UTC](https://forums.percona.com/t/pmm2-alerting-not-working-since-upgrade-to-2-31/18316/1 "2022-11-02T13:36:50Z")
I had an alert set up on MySQL Down, which sent me an email whenever a system went down. This has worked fine until fairly recently. I upgraded to 2.31 a week or so ago so I can only assume it’s related to this. Today a system went down and I didn’t get an email. I am unable to create new alerts - “Failed to save rule: Internal server error.; Internal server error.” on the webpage and: ```auto logger=context traceID=00000000000000000000000000000000 userId=5 orgId=1 uname=psumner t=2022-11-02T13:29:27.62533991Z level=error msg="Error from access control system" error="could not resolve datasources:id:1: data source not found" accessErrorID=ACE2718563073 logger=context traceID=00000000000000000000000000000000 userId=5 orgId=1 uname=psumner t=2022-11-02T13:29:27.625439734Z level=info msg="Request Completed" method=GET path=/api/datasources/1 status=403 remote_addr=127.0.0.1 time_ms=2 duration=2.428558ms size=166 referer= traceID=00000000000000000000000000000000 ``` This from grafana.log Any clues where to start looking? I’ve tried various created and default templates, tried creating alerts in existing folder, and also created a new folder for the alerts. Same issue. I’ve tried disabling and re-enabling Alerting in the settings menu and the problem persists. [Edit] Since writing this I’ve found the python script to migrate alerts. I’ve run it and… ```auto [root@rsnpmm01 ~]# python3 alertrulemigration.py -uadmin -padmin Request existing IA rules. Request existing IA rules done. There are no rules to migrate, exiting. ``` ---
**Author:** ![k3r1m](https://avatars.discourse-cdn.com/v4/letter/k/f9ae1b/32.png) [@k3r1m](https://forums.percona.com/u/k3r1m)\ **Post date:** [November 2, 2022, 3:36pm UTC](https://forums.percona.com/t/pmm2-alerting-not-working-since-upgrade-to-2-31/18316/2 "2022-11-02T15:36:41Z")
hi where can i find this python file . it is in the server please give to me .py file location or where can i find thank you. ---
**Author:** ![psumner](https://avatars.discourse-cdn.com/v4/letter/p/e8c25b/32.png) [@psumner](https://forums.percona.com/u/psumner)\ **Post date:** [November 2, 2022, 3:43pm UTC](https://forums.percona.com/t/pmm2-alerting-not-working-since-upgrade-to-2-31/18316/3 "2022-11-02T15:43:34Z")
> **[Percona Alerting - Percona Monitoring and Management](https://percona.com/percona-monitoring-and-management/using/alerting.html#migrate-alert-rules)** > > Documentation ---
**Author:** ![k3r1m](https://avatars.discourse-cdn.com/v4/letter/k/f9ae1b/32.png) [@k3r1m](https://forums.percona.com/u/k3r1m)\ **Post date:** [November 2, 2022, 3:58pm UTC](https://forums.percona.com/t/pmm2-alerting-not-working-since-upgrade-to-2-31/18316/4 "2022-11-02T15:58:01Z")
thank you for your answer but when i run this command in server ; python ia\_migration.py -u admin -p admin I get an error in bellow (python3: can’t open file ‘ia\_migration.py’: [Errno 2] No such file or directory) ---
**Author:** ![steve.hoffman](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/steve.hoffman/32/66_2.png) [@steve.hoffman](https://forums.percona.com/u/steve.hoffman)\ **Post date:** [November 2, 2022, 5:34pm UTC](https://forums.percona.com/t/pmm2-alerting-not-working-since-upgrade-to-2-31/18316/5 "2022-11-02T17:34:50Z")
@k3r1m you may need to do it like `python3 /path/to/ia_migration.py -u admin - p admin` @psumner I’m posting this internally to see if the team that works on alerting can offer some help. ---
**Author:** ![Harry\_Geffin](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/harry_geffin/32/4039_2.png) [@Harry\_Geffin](https://forums.percona.com/u/Harry_Geffin)\ **Post date:** [November 2, 2022, 7:06pm UTC](https://forums.percona.com/t/pmm2-alerting-not-working-since-upgrade-to-2-31/18316/6 "2022-11-02T19:06:40Z")
Hello psumner,It might be helpful to know - you upgraded from what previous version to 2.31.0? ---
**Author:** ![k3r1m](https://avatars.discourse-cdn.com/v4/letter/k/f9ae1b/32.png) [@k3r1m](https://forums.percona.com/u/k3r1m)\ **Post date:** [November 2, 2022, 8:37pm UTC](https://forums.percona.com/t/pmm2-alerting-not-working-since-upgrade-to-2-31/18316/7 "2022-11-02T20:37:00Z")
that’s the issue I do not know the path where is migration.py what can i find it ? ---
**Author:** ![steve.hoffman](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/steve.hoffman/32/66_2.png) [@steve.hoffman](https://forums.percona.com/u/steve.hoffman)\ **Post date:** [November 2, 2022, 8:42pm UTC](https://forums.percona.com/t/pmm2-alerting-not-working-since-upgrade-to-2-31/18316/8 "2022-11-02T20:42:15Z")
ahh, you have to download it here: > and then run the command from the folder you downloaded the file to. ---
**Author:** ![psumner](https://avatars.discourse-cdn.com/v4/letter/p/e8c25b/32.png) [@psumner](https://forums.percona.com/u/psumner)\ **Post date:** [November 3, 2022, 8:06am UTC](https://forums.percona.com/t/pmm2-alerting-not-working-since-upgrade-to-2-31/18316/9 "2022-11-03T08:06:36Z")
Oh I thought I’d included that. Previous version was 2.30 ---
**Author:** ![psumner](https://avatars.discourse-cdn.com/v4/letter/p/e8c25b/32.png) [@psumner](https://forums.percona.com/u/psumner)\ **Post date:** [March 8, 2023, 8:35am UTC](https://forums.percona.com/t/pmm2-alerting-not-working-since-upgrade-to-2-31/18316/10 "2023-03-08T08:35:03Z")
Is there anything I can do to completely reset the alerting system without defaulting everything else? It’s a real nuisance this not working. I’ve since upgraded to 2.35 and still get the same error on the webpage, but now seeing this error in the pmm-managed.log: ^[[36mINFO^[[0m[2023-03-08T08:47:11.916+00:00] Starting RPC /alerting.v1.Alerting/CreateRule … ^[[36mrequest^[[0m=d160a25b-bd8d-11ed-ac03-000c29e5ef7c ^[[31mERRO^[[0m[2023-03-08T08:47:11.926+00:00] RPC /alerting.v1.Alerting/CreateRule done in 10.299066ms with unexpected error: status: 404, body: {“message”:“Data source not found”} ^[[31mrequest^[[0m=d160a25b-bd8d-11ed-ac03-000c29e5ef7c ---
**Author:** ![psumner](https://avatars.discourse-cdn.com/v4/letter/p/e8c25b/32.png) [@psumner](https://forums.percona.com/u/psumner)\ **Post date:** [March 16, 2023, 2:08pm UTC](https://forums.percona.com/t/pmm2-alerting-not-working-since-upgrade-to-2-31/18316/11 "2023-03-16T14:08:03Z")
Right I’ve done some more digging around and it looks like I’ve somehow ended up with a datasource that isn’t ID #1 which everything assumes it will be > [ > { > “id”: 15, > “uid”: “ge\_Z6m-Vk”, > “orgId”: 1, > “name”: “Alertmanager”, > “type”: “alertmanager”, > “typeName”: “Alertmanager”, > “typeLogoUrl”: “public/app/plugins/datasource/alertmanager/img/logo.svg”, > “access”: “proxy”, > “url”: “”, > “user”: “”, > “database”: “”, > “basicAuth”: false, > “isDefault”: false, > “jsonData”: {}, > “readOnly”: false > }, > { > “id”: 12, > “uid”: “PT9pF\_Lnz”, > “orgId”: 1, > “name”: “ClickHouse”, > “type”: “vertamedia-clickhouse-datasource”, > “typeName”: “Altinity plugin for ClickHouse”, > “typeLogoUrl”: “public/plugins/vertamedia-clickhouse-datasource/img/altinity\_logo.svg”, > “access”: “proxy”, > “url”: “[http://127.0.0.1:8123](http://127.0.0.1:8123)”, > “user”: “”, > “database”: “”, > “basicAuth”: false, > “isDefault”: false, > “jsonData”: { > “keepCookies”: null > }, > “readOnly”: true > }, > { > “id”: 10, > “uid”: “0TrpFlYnk”, > “orgId”: 1, > “name”: “Metrics”, > “type”: “prometheus”, > “typeName”: “Prometheus”, > “typeLogoUrl”: “public/app/plugins/datasource/prometheus/img/prometheus\_logo.svg”, > “access”: “proxy”, > “url”: “[http://127.0.0.1:8430/prometheus/](http://127.0.0.1:8430/prometheus/)”, > “user”: “”, > “database”: “”, > “basicAuth”: false, > “isDefault”: true, > “jsonData”: { > “httpMethod”: “POST”, > “keepCookies”: null, > “timeInterval”: “1s” > }, > “readOnly”: true > }, > { > “id”: 11, > “uid”: “ao9tFlY7z”, > “orgId”: 1, > “name”: “PostgreSQL”, > “type”: “postgres”, > “typeName”: “PostgreSQL”, > “typeLogoUrl”: “public/app/plugins/datasource/postgres/img/postgresql\_logo.svg”, > “access”: “proxy”, > “url”: “localhost:5432”, > “user”: “postgres”, > “database”: “pmm-managed”, > “basicAuth”: false, > “isDefault”: false, > “jsonData”: { > “postgresVersion”: “1100”, > “sslmode”: “disable” > }, > “readOnly”: true > }, > { > “id”: 14, > “uid”: “uo9tK\_Ynz”, > “orgId”: 1, > “name”: “Prometheus AlertManager”, > “type”: “camptocamp-prometheus-alertmanager-datasource”, > “typeName”: “Prometheus AlertManager”, > “typeLogoUrl”: “public/plugins/camptocamp-prometheus-alertmanager-datasource/img/json-logo.svg”, > “access”: “proxy”, > “url”: “[http://localhost:9093/alertmanager/](http://localhost:9093/alertmanager/)”, > “user”: “”, > “database”: “”, > “basicAuth”: false, > “isDefault”: false, > “jsonData”: { > “keepCookies”: null > }, > “readOnly”: true > }, > { > “id”: 13, > “uid”: “8orpKlYnk”, > “orgId”: 1, > “name”: “PTSummary”, > “type”: “pmm-pt-summary-datasource”, > “typeName”: “PTSummary”, > “typeLogoUrl”: “public/img/icn-datasource.svg”, > “access”: “proxy”, > “url”: “”, > “user”: “”, > “database”: “”, > “basicAuth”: false, > “isDefault”: false, > “jsonData”: {}, > “readOnly”: true > } > ] I’ve mangled the Python script to copy rules and hard-coded rule 10 to be the source and blow me did that almost work. > Request existing IA rules. > Request existing IA rules done. > Found rules: 2. > Create alert group for migrated rules. > Create alert group for migrated rules done. > Get datasource UID. > Get datasource UID done. > Convert IA rules. > Convert IA rules done. > Send request to create migrated alerts. > Send request to create migrated alerts done. > Server response: > {“message”:“rule group updated successfully”} This is great, I’ve got my two previous alerts showing up - but I still can’t create new ones. Presumably datasource 1 is hardcoded somewhere. How do I fix this? ---
**Author:** ![psumner](https://avatars.discourse-cdn.com/v4/letter/p/e8c25b/32.png) [@psumner](https://forums.percona.com/u/psumner)\ **Post date:** [March 17, 2023, 9:47am UTC](https://forums.percona.com/t/pmm2-alerting-not-working-since-upgrade-to-2-31/18316/12 "2023-03-17T09:47:38Z")
One final update from me - I’ve worked out that I can create Grafana managed alerts so I have been copying details from the PMM alert templates to the Grafana Managed Alert pages. I can at least get things going again even if it’s not ideal. ---
**Author:** ![psumner](https://avatars.discourse-cdn.com/v4/letter/p/e8c25b/32.png) [@psumner](https://forums.percona.com/u/psumner)\ **Post date:** [March 17, 2023, 11:54am UTC](https://forums.percona.com/t/pmm2-alerting-not-working-since-upgrade-to-2-31/18316/13 "2023-03-17T11:54:59Z")
Final final final update - as it’s now working. > # sqlite3 /srv/grafana/grafana.db > […] > sqlite\> select \* from data\_source; > > 10|1|1|prometheus|Metrics|proxy|[http://127.0.0.1:8430/prometheus/||||0|||1|{“httpMethod”:“POST”,“keepCookies”:null,“timeInterval”:“1s”}|2022-03-09](http://127.0.0.1:8430/prometheus/%7C%7C%7C%7C0%7C%7C%7C1%7C%7B%22httpMethod%22:%22POST%22,%22keepCookies%22:null,%22timeInterval%22:%221s%22%7D%7C2022-03-09) 15:01:14|2023-03-16 14:23:43|0|{}|1|0TrpFlYnk > > sqlite\> .schema data\_source > > […] > > sqlite\> update data\_source set `id`=1 where `id`=10; I can now create PMM-Managed alerts using the UI. Everything else seems to have carried on working.