QAN not showing any data in PMM2 x Percona MySQL 8.0.34

[virin_t]

Description:

We have recently upgraded mysql to percona 8.0.34, We already have PMM2 working

However, I am unable to to view data under Query Analytics. What would be the cause and troubleshooting steps to fix or have QAN working.

also, we have another grafana board for monitoring, it gets data from prometheus from entire fleet on instances, in that board I am seeing CPU spike for the database server, but no spikes are shown on PMM for CPU panel.

[matthewb]

Hey @virin_t,
Which source are you using for QAN? When you ran pmm-admin add mysql? If you used slow query log, then queries will only be visible in QAN if 1) slow log is enabled in mysql, AND 2) a query takes longer than long_query_time to execute.

@virin_t

but no spikes are shown on PMM for CPU panel.

Do you have any data in CPU panel, or is it blank ‘No Data’? If no data, then the agent is likely installed incorrectly, or something is blocking PMM from getting the agent data. If you have data, be sure you are comparing the same time range with same resolutions.

[virin_t]

attaching

the two for reference

[matthewb]

Well, PMM is showing CPU %, while the other is showing CPU ‘K’ (what unit is that?) I would compare how PMM’s node exporter collects CPU info (/proc/stat) to how your other one is collecting data.

Why does ‘8.32K’ on the other equal to almost 100% CPU usage? Something doesn’t look right. What if your system does ‘9.92K’? Would that go > 100%?

[virin_t]

Not sure, I just want to make sure , which metrics are correct at CPU level.

and the board in PMM is showing accurately or is there anything I need to double check that we are getting correct metrics to PMM board.

attaching the screenshot from the other board (k)

[matthewb]

You’ll need to look at the source for the graph. Click the “3 dots” in your other graph, and you should see some option to view the JSON of the panel. Or maybe click ‘Edit’, to modify the graph. In either location, you’ll be able to see the metric names, and what unit the graph is plotting against. There could easily be a mistake in graph definition.

I can’t make any determination with what you’ve shown. Your graph shows some unknown ‘K’ unit, which somehow translates to near 100%. ‘K’ and ‘%’ are not the same.

Since I can see PMM’s source code, I know exactly where the data is coming from, and I know the unit of the graph, so I’m trusting PMM more at this time until shown otherwise.