For the first time to add an alarm, follow-up without newspaper police
we have two articles about email alerting in Grafana on our blog, feel free to use them
[url]https://www.percona.com/blog/2017/01/23/mysql-and-mongodb-alerting-with-pmm-and-grafana/[/url]
[url]https://www.percona.com/blog/2017/02/02/pmm-alerting-with-grafana-working-with-templated-dashboards/[/url]
you can follow instructions from the first link.
is “Send Test” button working fine for you?
Send Test normal,but only for the first time
Am I understand correctly that “Send Test” button become broken after some time?
Do you see any errors in grafana log file when button don’t send email?
you can open log file with the following command:
docker exec -it pmm-server less /var/log/grafana/grafana.log
t=2017-06-14T00:02:40+0000 lvl=eror msg=“Alert Rule Result Error” logger=alerting.evalHandler ruleId=5 name=“CPU Usage alert” error=“tsdb.HandleRequest() error bad_response: bad response code 400” changing state to=alerting
t=2017-06-14T00:03:01+0000 lvl=eror msg=“Alert Rule Result Error” logger=alerting.evalHandler ruleId=1 name=“MySQL Active Threads alert” error=“tsdb.HandleRequest() error bad_response: bad response code 400” changing state to=alerting
t=2017-06-14T00:03:20+0000 lvl=eror msg=“Alert Rule Result Error” logger=alerting.evalHandler ruleId=2 name=“Connections alert” error=“tsdb.HandleRequest() error time: invalid duration $interval” changing state to=alerting
t=2017-06-14T00:03:20+0000 lvl=eror msg=“Alert Rule Result Error” logger=alerting.evalHandler ruleId=5 name=“CPU Usage alert” error=“tsdb.HandleRequest() error bad_response: bad response code 400” changing state to=alerting
t=2017-06-14T00:04:01+0000 lvl=eror msg=“Alert Rule Result Error” logger=alerting.evalHandler ruleId=1 name=“MySQL Active Threads alert” error=“tsdb.HandleRequest() error bad_response: bad response code 400” changing state to=alerting
t=2017-06-14T00:04:20+0000 lvl=eror msg=“Alert Rule Result Error” logger=alerting.evalHandler ruleId=2 name=“Connections alert” error=“tsdb.HandleRequest() error time: invalid duration $interval” changing state to=alerting
t=2017-06-14T00:04:40+0000 lvl=eror msg=“Alert Rule Result Error” logger=alerting.evalHandler ruleId=5 name=“CPU Usage alert” error=“tsdb.HandleRequest() error bad_response: bad response code 400” changing state to=alerting
t=2017-06-14T00:05:01+0000 lvl=eror msg=“Alert Rule Result Error” logger=alerting.evalHandler ruleId=1 name=“MySQL Active Threads alert” error=“tsdb.HandleRequest() error bad_response: bad response code 400” changing state to=alerting
t=2017-06-14T00:05:20+0000 lvl=eror msg=“Alert Rule Result Error” logger=alerting.evalHandler ruleId=2 name=“Connections alert” error=“tsdb.HandleRequest() error time: invalid duration $interval” changing state to=alerting
t=2017-06-14T00:05:20+0000 lvl=eror msg=“Alert Rule Result Error” logger=alerting.evalHandler ruleId=5 name=“CPU Usage alert” error=“tsdb.HandleRequest() error bad_response: bad response code 400” changing state to=alerting
t=2017-06-14T00:06:01+0000 lvl=eror msg=“Alert Rule Result Error” logger=alerting.evalHandler ruleId=1 name=“MySQL Active Threads alert” error=“tsdb.HandleRequest() error bad_response: bad response code 400” changing state to=alerting
t=2017-06-14T00:06:20+0000 lvl=eror msg=“Alert Rule Result Error” logger=alerting.evalHandler ruleId=2 name=“Connections alert” error=“tsdb.HandleRequest() error time: invalid duration $interval” changing state to=alerting
t=2017-06-14T00:06:40+0000 lvl=eror msg=“Alert Rule Result Error” logger=alerting.evalHandler ruleId=5 name=“CPU Usage alert” error=“tsdb.HandleRequest() error bad_response: bad response code 400” changing state to=alerting
t=2017-06-14T00:07:02+0000 lvl=eror msg=“Alert Rule Result Error” logger=alerting.evalHandler ruleId=1 name=“MySQL Active Threads alert” error=“tsdb.HandleRequest() error bad_response: bad response code 400” changing state to=alerting
t=2017-06-14T00:07:20+0000 lvl=eror msg=“Alert Rule Result Error” logger=alerting.evalHandler ruleId=2 name=“Connections alert” error=“tsdb.HandleRequest() error time: invalid duration $interval” changing state to=alerting
I see “” issue in the logs
It looks like internal grafana issue, see https://github.com/grafana/grafana/issues/8353
the latest message from grafana developers on github
so it is needed to update grafana and resave alert. With grafana update we have two options:
- wait a long time for PMM 1.2.0 release.
- update grafana manually (i can prepare instructions for you)
what do you think? do you want to wait for 1.2.0 or run some update commands manually?