CPU/memory/IO/network metrics not updating after 2.12 upgrade

Percona Monitoring and Management (PMM) PMM 2.x
1

Hi,

the graphs for most host metrics do not show new data after upgrading to 2.12.

Amazon Aurora / MySQL 5.6, nodes are added via the API /v1/management/RDS/Add

{

“region”: “REGION”,

“az”: “AZ”,

“instance_id”: “INSTANCE_ID”,

“node_model”: “NODE_MODEL”,

“address”: “ADDRESS”,

“port”: 3306,

“engine”: “ENGINE”,

“node_name”: “INSTANCE_NAME”,

“cluster”: “CLUSTER_IDENTIFIER”,

“username”: “PMM_DB_USER”,

“password”: “PMM_DB_PASSWORD”,

“rds_exporter”: true,

“qan_mysql_perfschema”: true,

“tablestats_group_table_limit”: 3000,

“tls”: true,

“tls_skip_verify”: true

}

Untitled Image
Untitled Image815×226 9.78 KB

Untitled Image
Untitled Image835×225 14.5 KB

2

Hi @scf ,

We are monitoring Amazon Aurora / MySQL 5.7 on our pmmdemo installation.

Disk IO Load data is still shown.

Could you provide us with other disappeared metrics?

https://pmmdemo.percona.com/graph/d/node-instance-summary/node-summary?viewPanel=51&orgId=1&var-node_name=rds-aurora57-instance-1

Screenshot_20201218_105742.png
Screenshot_20201218_105742.png1917×851 69 KB

3

Hi,

looks like all RDS enhanced metrics are affected, the /enhanced endpoint is not returning any metrics for those nodes. (config looks similar, except the affected nodes do not have aws_access_key set as they are using the instance profile/role)

I deleted some older nodes from the PMM inventory (using the UI), however, the rds_exporter config wasn’t updated (/tmp/rds_exporter/pmm-server/rds/config).

There are errors in the pmm-agent.log: “Failed to filter log events: ResourceNotFoundException: The specified log stream does not exist.”

The upgrade was from 2.10.1 directly to 2.12.

4

I suspect it’s this known bug: https://jira.percona.com/browse/PMM-6490

https://github.com/percona/rds_exporter/pull/47

5

after some tests it actually seems to be this old bug, if the “RDSOSMetrics” log stream for an instance isn’t found (e.g. due to disabled enhanced monitoring), it will not collect metrics for the other instances:

https://jira.percona.com/browse/PMM-3588