After migration all rules are lost with the following message now showing the following
Errors loading rules
Docker upgrade via the gui
2.39 to 2.40
logs.txt (66.5 KB)
Rules Should remain
Alert Rules were lost
Hello @meyerder
In this case, you need to create/convert alert templates as the new pmm version changed many things for alerting.Percona Alerting - Percona Monitoring and Management
If you are upgrading from PMM 2.25 and earlier, alert templates will not be automatically migrated. This is because PMM 2.26.0 introduced significant changes to the core structure of rule templates.
In this scenario, you will need to manually recreate any custom rule templates that you want to transfer to PMM 2.26.0 or later.
Alert rules created with Integrated Alerting in PMM 2.30 and earlier are not automatically migrated to Percona Alerting.
@lalit.choudhary ALL alert rules are gone… All Alert templates migrated properly… All Alert Rules/templates were built on 2.39 as I have not had this instance up for more than a month or maybe 45 days
If you could give me a clue as to where to look to troubleshoot the issue that would help
I found this… but that is sqllite and not PostgreSQL It makes me think that the issue might be with one of the grafana alerts that may have existed (I cant remember but I may have had one or two grafana managed alerts vs all pmm templates)
opened 02:11PM - 15 Apr 23 UTC
needs investigation
<!--
Please use this template to create your bug report. By providing as much i… nfo as possible you help us understand the issue, reproduce it and resolve it for you quicker. Therefore take a couple of extra minutes to make sure you have provided all info needed.
PROTIP: record your screen and attach it as a gif to showcase the issue.
- Questions should be posted to: https://community.grafana.com
- Use query inspector to troubleshoot issues: https://bit.ly/2XNF6YS
- How to record and attach gif: https://bit.ly/2Mi8T6K
-->
**What happened**:
**Switching from sqlite3 to mysql**, unable to show old alert rules:
"Failed to load Grafana rules state: failure getting rules: invalid character 'c' after object key:value pair; failure getting rules: invalid character 'c' after object key:value pair
Failed to load Grafana rules config: failed to get alert rules: invalid character 'c' after object key:value pair"
**Even when I create **a new alert rule****, it shows:"Failed to save rule: failed to get group alert rules: invalid character 'c' after object key:value pair"
**Datasource**: Prometheus
**Metrics browser**: kube_pod_container_status_ready{namespace=~"aaa|bbb"}
**B expression**:
Operation Reduce, Function:Last, Input:A, Mode:Strict
**C expression**
Operation:Math, Expression: $B > 0
**Evaluate every**: 10s for 30s
**Folder**：“xxx”， GROUP：“COMMON”
** grafana log **
logger=context traceID=00000000000000000000000000000000 userId=1 orgId=1 uname=admin t=2023-04-15T22:09:15.061363967+08:00 level=error msg="failed to get alert rules: invalid character 'c' after object key:value pair" error="failed to get alert rules: invalid character 'c' after object key:value pair" remote_addr=10.249.102.15 traceID=00000000000000000000000000000000
logger=context traceID=00000000000000000000000000000000 userId=1 orgId=1 uname=admin t=2023-04-15T22:10:08.129559456+08:00 level=error msg="failed to get group alert rules: invalid character 'c' after object key:value pair" error="failed to get group alert rules: invalid character 'c' after object key:value pair" remote_addr=10.249.102.15 traceID=00000000000000000000000000000000
logger=context traceID=00000000000000000000000000000000 userId=1 orgId=1 uname=admin t=2023-04-15T22:10:08.129761725+08:00 level=error msg="Request Completed" method=GET path=/api/ruler/grafana/api/v1/rules/运营监控/COMMON status=500 remote_addr=10.249.102.15 time_ms=30 duration=30.098243ms size=141 referer="http://10.14.32.240:3000/alerting/new?returnTo=%2Falerting%2Flist" traceID=00000000000000000000000000000000
logger=ngalert t=2023-04-15T22:10:10.003059703+08:00 level=error msg="scheduler failed to update alert rules" err="failed to get alert rules: invalid character 'c' after object key:value pair"
**What you expected to happen**:
can load old alert rules and save alert rule normally.
**How to reproduce it (as minimally and precisely as possible)**:
everytime when I create this alert rule.
**Anything else we need to know?**:
Migaration from sqllite3 to mysql 5.7.29.
**Environment**:
- Grafana version: v9.1.0
- Data source type & version: Prometheus
- OS Grafana is installed on: Ubuntu
- User OS & Browser: Chorme
- Grafana plugins:
- Others: Using mysql(5.7.29) for storing
nurlan
October 20, 2023, 11:32am
4
@meyerder could you run following commands within docker container and provide resultpsql -U grafanaselect id, data, from alert_rule
@nurlan Looks like it was a single alert rule… the data was not in a proper json format it looks like… I deleted this rule and all works…
[{"refId":"A","queryType":"range","relativeTimeRange":{"from":3600,"to":0},"datasourceUid":"NuQ5hK64k","model":{"datasource":{"type":"loki","uid":"NuQ5hK64k"},"editorMode":"builder","expr":"sum by(applid) (count_over_time({job=\"system_logs\", filename=\"/var/log/messages\", applid=~\"xxx\", hostname=\"xxxxx\"} |~ "xxx" [1h]))","hide":false,"intervalMs":1000,"legendFormat":"{{appid}}","maxDataPoints":43200,"queryType":"range","refId":"A"}},{"refId":"B","queryType":"","relativeTimeRange":{"from":3600,"to":0},"datasourceUid":"__expr__","model":{"conditions":[{"evaluator":{"params":[14],"type":"lt"},"operator":{"type":"and"},"query":{"params":["A"]},"reducer":{"params":[],"type":"avg"},"type":"query"}],"datasource":{"type":"__expr__","uid":"__expr__"},"expression":"A","hide":false,"intervalMs":1000,"maxDataPoints":43200,"refId":"B","type":"classic_conditions"}}]
