"No data" on PXC/Galera Cluster Summary

Percona Monitoring and Management (PMM) PMM 1.x
1

After updating the PMM server docker instance from 2.1.0->2.2.0 I noticed that all graphs in this dashboard no longer show any data.

I have also updated the clients on all of the systems monitored. Interestingly enough, before the update the dashboard “MySQL Command/Handler counters compare”
reported “No Data” on all graphs except for one host. This was auto-resolved after this update.

However, everything else so far seems to be fine.

Any clues to what might be going on here?

2

Hi milosb . Thanks for staying on top of our new version and reporting problems!
Have you used some Labels (cluster, environment) for your installation? if yes - can you provide them?

Also pls check https://PMM_SERVER/prometheus/targets for Errors

3

Hi roma.novikov No, not that I’m aware of. Please remind me where I’ve seen the option to add Labels, just so I know we’re on the same page.

As for the Prometheus targets, everything is fine there - that’s the first thing I check. No unhealthy services.

I’ve moved this environment from PMMv1 over to PMMv2 back in October and it worked well - except for the “No Data” dashboards,… earlier with the MySQL Handler/command counters, now this.

4

milosb you can see your labels on https://PMM_SERVER/graph/d/pmm-inventory/pmm-inventory “Services” tab

5

roma.novikov There’s just a table in there with all the nodes monitored,… with columns: ID, Service Type, Service Name, Node ID, Addresses, Port, Other details

Service name is in the form ${HOSTNAME}-mysql

Is there something else I should be lookin at?

6

The awkward thing is: Before the update to v2.2.0 (and even while I was on 2.0.1) there was “No Data” in the MySQL Handlers/Command Counters shown… but after the update, they’re all of the sudden there - even for the previous months. So, the data was collected, but for some reason not shown.

I suspect it’s the same situation now, but am unable to figure out where the problem lies. The Grafana log shows only this, related to alert graph images not attached when alerts are sent (this also used to work:

lvl=eror msg=“Failed to upload alert panel image.” logger=alerting.notifier error="Timeout error. You can set timeout in seconds with &timeout url parameter

7

I’ll also add that the PXC/Galera Node Summary and PXC/Galera Nodes Compare display data just fine for all cluster nodes.

The drop-down menu in the PXC/Galera Cluster Summary shows no cluster… Only “None” to choose from.

8

roma.novikov Here’s some development of this issue: If I open the PXC/Galera Node summary dashboard, the graphs display fine both initially and after
a refresh interval, while the “Cluster” drop-down menu has a cluster name to choose from, as it was set in the 'wsrep_cluster_name".
Switching from this, to the PXC/Galera Cluster Summary initially loads the graphs fine, but after the very first refresh the graphs are gone, with “No data” filling the place of the
curves… and there’s no more value to choose from in the “Cluster” drop-down menu.
Hitting the Cluster menu before the refresh, again results in “No data” in the graphs, and no options to choose from.

Entering the cluster name manually in the drop-down menu brings the graphs back up again, but again only until the first dashboard refresh. Looks like a bug…

Hope this helps.