# Alerts taking too long to be sent on repeating events.

**URL:** <https://forums.percona.com/t/alerts-taking-too-long-to-be-sent-on-repeating-events/7532>\
**Category:** PMM 2.x\
**Tags:** pmm\
**Created:** [April 12, 2020, 3:31pm UTC](https://forums.percona.com/t/alerts-taking-too-long-to-be-sent-on-repeating-events/7532 "2020-04-12T15:31:42Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![jbiancot](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/jbiancot/32/6030_2.png) [@jbiancot](https://forums.percona.com/u/jbiancot)\
**Post date:** [April 12, 2020, 3:31pm UTC](https://forums.percona.com/t/alerts-taking-too-long-to-be-sent-on-repeating-events/7532/1 "2020-04-12T15:31:42Z")

</div>

Hi folks,  
I grabbed the docker image: percona/pmm-server, v2.4.0, then I played with it today for a while. Then, I setup the SMTP inside the docker container, I configured a MySQL server to report the PMM Server, etc. So far, so good.  
  
Then, I have been stopping and starting the MySQL services and taking the time when it fires 1st alert, when it will send the recovered e-mail, and so.  
  
I am playing with the “Evaluate every 1m For 0m”. It realizes in 2 minutes, more of less of the 1st MySQL service down situation, but after a few seconds I am starting the MySQL service, let it run for a few minutes, then I stop the MySQL service. It takes over 5 minutes to sent the 2nd alert letting me know the server is down.  
  
The query I am using:  
avg by (service\_name) (mysql\_global\_status\_uptime{service\_name=~“service-mysql”}), and I setup like a month.  
What I am trying to archive, if the condition “no data or on error”, send me the alert.  
Also, I am not seeing a 2nd alert “like a reminder” service is down.  
Any idea how to make it more resilient ?  
 The goal will be: if server/service goes down, then an e-mail should be sent almost immediately “on a 1 minute”, and perhaps after 2 minutes a reminder should be sent. After the MySQL service gets recovered, it should send an e-mail about it, AND if after a few seconds the services goes down “again”, it resets: and it sends a new service down email.  
Cheers,

---

<div class="post-metadata">

**Author:** ![jbiancot](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/jbiancot/32/6030_2.png) [@jbiancot](https://forums.percona.com/u/jbiancot)\
**Post date:** [April 12, 2020, 4:25pm UTC](https://forums.percona.com/t/alerts-taking-too-long-to-be-sent-on-repeating-events/7532/2 "2020-04-12T16:25:09Z")

</div>

Here is a screen shot of when it realized I stopped and started the MySQL service. The recovery took 1-2 minutes, to alert but it took up to 5-6 minutes in realize the service went down. When I received the recovery alert, I waited between 30-60 secs to stop the MySQL service, and it took way to long to realize service wen down and alert.

 ![Screen Shot 2020-04-12 at 11.52.39 PM.png](https://us1.discourse-cdn.com/flex019/uploads/percona1/original/2X/1/1256f4e43acff7545b294e95bd9d251df9613981.png)

---

<div class="post-metadata">

**Author:** ![jbiancot](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/jbiancot/32/6030_2.png) [@jbiancot](https://forums.percona.com/u/jbiancot)\
**Post date:** [April 16, 2020, 11:08pm UTC](https://forums.percona.com/t/alerts-taking-too-long-to-be-sent-on-repeating-events/7532/3 "2020-04-16T23:08:25Z")

</div>

Anybody know about this?

---

<div class="post-metadata">

**Author:** ![Cihan\_Tunali](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/cihan_tunali/32/4574_2.png) [@Cihan\_Tunali](https://forums.percona.com/u/Cihan_Tunali)\
**Post date:** [April 17, 2020, 2:32am UTC](https://forums.percona.com/t/alerts-taking-too-long-to-be-sent-on-repeating-events/7532/4 "2020-04-17T02:32:39Z")

</div>

Im doing same thing to backup my mongodb. ım using cronjob to stop mongodb, start snapshot, start mongodb. I sensed that, after starting service pmm-agent could not get the state. I found a workaround with restarting pmm-agent service after starting mongodb. you can try with this way.

---

<div class="post-metadata">

**Author:** ![jbiancot](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/jbiancot/32/6030_2.png) [@jbiancot](https://forums.percona.com/u/jbiancot)\
**Post date:** [April 18, 2020, 6:25am UTC](https://forums.percona.com/t/alerts-taking-too-long-to-be-sent-on-repeating-events/7532/5 "2020-04-18T06:25:52Z")

</div>

> [Ghan](https://forums.percona.com/profile/Ghan) said: Im doing same thing to backup my mongodb. ım using cronjob to stop mongodb, start snapshot, start mongodb. I sensed that, after starting service pmm-agent could not get the state. I found a workaround with restarting pmm-agent service after starting mongodb. you can try with this way.

  
I am not sure if We are on the same page. The goal is the PMM alerts are constant. E.g: a minute after one of the services (MySQL, MongoDB) doesn’t send their data then PMM will send the alert “email”. I have stopped and started just to test the behavior of PMM and it is not consistent.

---

<div class="post-metadata">

**Author:** ![\_Joe\_Keilty](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/_joe_keilty/32/17804_2.png) [@\_Joe\_Keilty](https://forums.percona.com/u/_Joe_Keilty)\
**Post date:** [September 11, 2024, 10:38am UTC](https://forums.percona.com/t/alerts-taking-too-long-to-be-sent-on-repeating-events/7532/6 "2024-09-11T10:38:39Z")

</div>

I think this is managed by the Notification Policy ‘Group Interval’ setting which defaults to 5m

“The waiting time to send a batch of new alerts for that group after the first notification was sent. Default 5 minutes.”
