Prometheus error

Hello,
Sometimes I got “Database is locked” and “context deadline exceeded” errors and all my alarms were triggered. How can I solve this?

It is PMM 2.9

Hello,

Anybody have an idea?

1 Like

Hi @Ghan could you add more context, please?
What have you done and how did you add alerts?
As far as I remember we didn’t support alerts in PMM 2.9.

1 Like

Hello Nurlan,
I am using grafana alerts and all alarm definitions are working well. But sometimes I got above errors from prometheus. Any idea?

1 Like

Well the “context deadline exceeded” is a timeout error message which I mostly see when there is no answer in the timeout period so depending on your system load it’s possible the query that the alert is running took longer than the timeout (dont’t recall if it’s 30 of 60 seconds). and assuming the second message is related to the first it could simply be too much contention on the datasource (prometheus db)… Are there a lot of users on this PMM instance or are the users doing big range data display?

1 Like

Thank you @steve.hoffman for your comment. I think you are right. I was using grafana with 5sn refresh period. I changed it to 30sn then my problems are gone. Not much users are using it and I am using it to show last 6 hours.

Anyway,I think i need to upgrade PMM to latest to check all my existing problem again. Thanks!

1 Like