I’ve read countless posts and articles but nothing seems to help me to find the problem.
I add an instance and it seems to start working but 12hrs or 24hrs later, everything goes to ‘no data’.
My only option to date is to remove everything and start over which is not doing much good.
Even when I remove the agent from services/agents, the server still shows up in the dashboard but not the mysql stats of course.
I’m not sure what info to share in this case but happy to share what ever you need if someone can help.
1 Like
Can you give some details on your setup? to see no data means 1 of 2 things…actually no data which is scary! That would mean the victoria metrics (assuming you’re referring to most panels) or clickhouse (if you mean Query analytics) lost it’s data. I also see this if my date range is set to something that actually has no data (I was using an old link on a system I redid…banged my head for some time until I realized it was a URL with an invalid date range hard coded in it).
When you get to the point of “No Data”, can you take a look at https:///victoriametrics/targets and click on ‘unhealthy’…anything jumping out there?
Did you setup a pmm-data volume? if not are you stop/starting the docker container?
What version of PMM are you using?
On your client, is there anything jumping out when you run pmm-admin status. if the time sync between client and server gets too far off that can break sending metrics (see that most when server is on a vm but a dead CMOS battery can also play games with the clock)
Any errors in your client syslog or if you run docker exec -it <pmm-server> tail -f /srv/log/pmm-managed.log?
Just some stuff that came to mind but with more detail I’m sure we’ll get there.
1 Like
Hi, sorry for the delay in replying but I’m at a bit of a loss.
I did what I was usually doing, deleting the instance and re-creating it.
The only difference this time is that I went to the Overview page in a different way.
Meaning that before yesterday, I would go to the PMM Dashboard, mysql, overview and eventually would see ‘No data’ on any of the reports.
Yesterday, the only change I made was to go to PMM dashboard, then I picked the node summary after mousing over the icon.
Now there seems to be data 24hrs later so I’m not sure at all what changed.
I’m running for vmware pmm-server-2.27.0.ova on esx 6.7.
I did not set up a pmm-data volume that I know of, only installed the vm, a few basic config things like time zone then I added an instance.
Depending on what I’m starting to use, I usually like to see it in action in the most basic way then I start reading documentation on the different sections, features, etc.
In victoriametrics/targets, clicking on unhealthy, just a blank white page.
In the All section, there is a lot of data and I don’t see anything that is looking like an error, just details on last polls etc.
Wish I knew what I did differently because now I feel like I’m just a click away from breaking it.
1 Like
seeing nothing on the unhealthy targets page is a good thing!!! means client(s) and server can see/talk to each other.
So if I’m understanding correctly: two different routes to the same “MySQL Instances Overview” give different results? Can you reproduce that on the fly? i.e. follow PMM Dashboard → Mysql → Ovierview and see no data but then Services → Mysql Instance overview and see the data. If this was the case I’d be highly suspicious about a bad link in one of the flows that’s setting the date range to something other than say “Last 12 hours” or possibly setting some filter in the URL.
or you’re saying everything is working?
Don’t worry about the data volume, that’s only needed with docker containers since the container is ephemeral and state isn’t preserved across certain actions without a dedicated data volume mounted.
I typically just run docker containers but I have vmware fusion so I’ll try it on my machine for 24 hours and see if I can reproduce it.
1 Like
Hi Steve,
Since posting this, I can see the stats here.
https://pmm.phx.loc/graph/d/mysql-instance-overview/mysql-instances-overview
This shows the DB stats.
I then moved away to
https://pmm.phx.loc/graph/d/mysql-instance-summary/mysql-instance-summary
and got a templating error but it went away too fast.
This shows a lot more details then the above.
However, under Node Summary, no data.
I don’t know really what happened that is different. Now I get ongoing data even when moving away from pages which was not the case when I first started.
Obviously, mystery problems are the worse. I could look at the server logs but I’m guessing I would find too much to post/share.
Maybe I need to just keep using it and see if it happens again?
1 Like
I’m going to ping a few teams internally to see if anyone has an idea…there should be no mysteries!
There are different exporters for Node stats vs DB stats (mysqld|postgresql|mongodb). The only time you wouldn’t normally see node stats is if you setup remote monitoring (as opposed to agents running on each client you want to monitor) where we’re fetching DB specific data right from the DB service port.
If you come up with any other details post them here otherwise we’ll go off what we know here!
1 Like
I’m very new to pmm so will always assume I did something wrong.
However, if you feel it’s important enough to want to dig/confirm anything, I’m happy to provide logs or some other info in PM.
Sorry I wasn’t able to provide better information but certainly appreciate your help.
1 Like