strange new behavior in QAN in pmm-admin 1.5.0

Percona Monitoring and Management (PMM) PMM 1.x
1

version: pmm-admin 1.5.0
Go Version | 1.9.2
PMM Network Status OK

I didn’t think twice of upgrading as the new graphs felt like they were targeted to me (RDS). Things looked good until I went into the query metrics. These are the problems:
pmm-admin list has these mysql:queries
mysql:queries RDS2 - YES pmm:@tcp(XXX:####) query_source=perfschema, query_examples=true
mysql:queries RDS3 - YES pmm:@tcp(XXX:####) query_source=perfschema, query_examples=true
mysql:queries RDS1 - YES pmm:***@tcp(XXX:####) query_source=perfschema, query_examples=true

but in the _pmm-query-analytics URL I have 6 hosts. These include the mongoDB (on ec2) and an RDS Postgresql instance.
mysql:queries RDS2 & mysql:queries RDS3 Seem to have the same data, even tho they have different databases & tables.
the PG connection has the same data as RDS2 & RDS3, as well as the mongo connection. RDS1 seems to have mostly it’s own queries but there are no longer examples. Most of the time it’s says the list of tables is empty (no db name). OR more disturbing it references a DB on RDS2.

let me know if you need more info.

2

Server and client: pmm-admin 1.5.1

Finding my qan instance id numbers was slightly different in this version but I managed to get it. My preferred cmd:
curl -s ‘http://10.xxx.xxx.xxx/v1/internal/ui/nodes?dc=dc1’ | python -mjson.tool | less
I found a BUNCH of pmm-admin remove mysql:queries & mongodb:queries. so I used the consul command to remove them (needed the qan UUID numbers to do this:
curl -X “DELETE” ‘http://10.xxx.xxx.xxx/qan-api/instances/’
They disappeared from the curl that checks.

BUT the _PMM Query Analytics shows all these servers in the drop down host (the ones removed above), many of them are not in the pmm-admin list.
and only 2 of the mysql servers that have been setup for mysql:queries have real queries from their RDS instances.

When I select a “phantom” server (one is a mongodb server) the queries are all a subset of the queries from one of the mysql:queries RDS instances that have been setup.

These “phantom” hosts are no long in
[url]http://xxx.xxx.xxx.xxx/prometheus/targets[/url]

So far not a fan of the new qan.

3

I have now updated to PMM server version 1.5.2. The pmm-admin is still at 1.5.1 as there is no 1.5.2.
Now I have NO queries for ANY hosts & I get this error (remember I used to have full access to queries for mysql and used this all the time. NOTHING has changed in the db)

error I got:
There is no query data because the MySQL Server is not been configured for monitoring. For details about the required configuration, see Configuring MySQL for Percona Monitoring and Management in PMM documentation.

Metrics still work.

I’m on RDS (both 5.6 and 5.7)

4

The setup_consumers:
select * from setup_consumers;
±---------------------------------±--------+
| NAME | ENABLED |
±---------------------------------±--------+
| events_stages_current | NO |
| events_stages_history | NO |
| events_stages_history_long | NO |
| events_statements_current | YES |
| events_statements_history | YES |
| events_statements_history_long | NO |
| events_transactions_current | NO |
| events_transactions_history | NO |
| events_transactions_history_long | NO |
| events_waits_current | NO |
| events_waits_history | NO |
| events_waits_history_long | NO |
| global_instrumentation | YES |
| thread_instrumentation | YES |
| statements_digest | YES |
±---------------------------------±--------+