I have created a Graph on my dashboard, which shows if the state of mongodb switches from PRIMARY to SECONDARY and from SECONDARY back to PRIMARY. And another query to notify if state is moving to RECOVERING.
There are basically 2 queries on this Alert which I am using for the above purpose.
A. (mongodb_mongod_replset_member_state{cluster=“mobilinedb”,instance=“GauravTest32165”,job=“mongodb”,name=“192.168.0.4:32165”,set=“mobilinedb”,state=“PRIMARY”}) or (mongodb_mongod_replset_member_state{cluster=“mobilinedb”,instance=“GauravTest32165”,job=“mongodb”,name=“192.168.0.4:32165”,set=“mobilinedb”,state=“SECONDARY”})
B. mongodb_mongod_replset_member_state{cluster=“mobilinedb”,instance=“GauravTest32165”,job=“mongodb”,name=“192.168.0.4:32165”,set=“mobilinedb”,state=“RECOVERING”}
I have attached my ALERT tab details as image files.
Also, have attached another image for a graph which overlaps.
I am having few concerns here:
- Is there any problems with my query, if Yes, what is the solution.
- When my mongodb node switches between PRIMARY and SECONDARY, I observed the graph overlaps for SECONDARY and PRIMARY both. Why is this happeneing? Is there an issue with Prometheus data source or issue with Grafana.
- If you observe another thing on the graph, considering my ALERT configuration, The small Icon “heart” shaped beside the Panel Title, it should be showing as GREEN. Whereas it is WHITE. why? how can we fix it?
Let me know, if you need any further details.
