Abnormal memory usage

[youngjae7732]

Hello.

I am monitoring the rds of aws as pmm.
So the mysqld-export process is running on the Monitoring Server.
However, the process occasionally uses more than 4 g of memory.
And if the process is not using memory for a certain period of time, it will not be able to collect data.
And if this is prolonged, it will lead to a Monitoring Server failure and restart the docker.
So I had a holiday yesterday, but I have not collected data for a long time.

Below is the relevant information.

ps -aux | grep mysqld_exporter :
root 32579 4.5 58.5 11134136 4576604 ? Dl 5월01 28:00 /usr/local/percona/pmm-client/mysqld_exporter -collect.auto_increment.columns -collect.binlog_size

Top : attach

data-graph : attach

Thank you.

[youngjae7732]

I am investigating the log after restart and I am getting an error in the log.
pmm-qan-agent, pmm-rds_exporter, consul logs.
The error contents are as follows.

consul.log :
2018/05/02 07:56:31 [WARN] consul: skipping reconcile of node {95f47f76e134 127.0.0.1 8301 map[build:unknown’’:2c77151 role:consul vsn:2 vsn_max:3 port:8300 wan_join_port:8302 vsn_min:2 raft_vsn:2 bootstrap:1 dc:dc1 id:8ac9d379-0677-9711-bc82-0c05f506b5b1] alive 1 5 2 2 5 4}

pmm-qan-agent.log :
2018/05/02 07:56:56.775450 WARNING data-sender Waiting for API to ack qan_1525236121454356084: read tcp 127.0.0.1:38432->127.0.0.1:80: i/o timeout
2018/05/02 07:57:04.776455 WARNING data-sender Waiting for API to ack qan_1525236121454356084: read tcp 127.0.0.1:38548->127.0.0.1:80: i/o timeout
2018/05/02 07:57:16.790024 WARNING data-sender Waiting for API to ack qan_1525236240700478029: read tcp 127.0.0.1:38726->127.0.0.1:80: i/o timeout
2018/05/02 07:57:25.024107 INFO mrms-monitor mysql instance 4d16bd9d4a0444e05cf4b189e6af366c restarted
2018/05/02 07:57:25.502403 INFO qan-analyzer-mysql-4d16bd9d First interval begins in 34.5 seconds
2018/05/02 07:57:25.534831 WARNING instance-manager PUT http://127.0.0.1/qan-api/instances/4d16bd9d4a0444e05cf4b189e6af366c failed: code 409:
2018/05/02 07:57:55.684788 WARNING data-sender Waiting for API to ack qan_1525236240700478029: read tcp 127.0.0.1:39422->127.0.0.1:80: i/o timeout
2018/05/02 07:58:00.000492 WARNING qan-analyzer-mysql-4d16bd9d-worker Interval out of sequence: got 1, expected 9
2018/05/02 07:58:03.685863 WARNING data-sender Waiting for API to ack qan_1525236240700478029: read tcp 127.0.0.1:39504->127.0.0.1:80: i/o timeout
2018/05/02 07:58:16.530204 WARNING data-sender Waiting for API to ack qan_1525236361807531143: read tcp 127.0.0.1:39690->127.0.0.1:80: i/o timeout

pmm-rds_exporter.log :
time=“2018-05-02T07:39:31Z” level=error msg=“unable to get logs for instance homedaymall-db-new: ResourceNotFoundException: The specified log group does not exist.\n\tstatus code: 400, request id: f3b82bb7-4ddb-11e8-b217-851b9339ad6b” source=“enhanced.go:108”
time=“2018-05-02T07:39:41Z” level=error msg=“unable to get logs for instance homedaymall-db-new: ResourceNotFoundException: The specified log group does not exist.\n\tstatus code: 400, request id: f9acd4b9-4ddb-11e8-8c5b-139d2f3a0334” source=“enhanced.go:108”
time=“2018-05-02T07:39:51Z” level=error msg=“unable to get logs for instance homedaymall-db-new: ResourceNotFoundException: The specified log group does not exist.\n\tstatus code: 400, request id: ffacc791-4ddb-11e8-ac1e-d7216051232a” source=“enhanced.go:108”

Could you tell me why this error is happening?

Thank you.

[lorraine.pocklington]

Thank you for the update youngjae7732

Would you be able to send us the 1 day and 7 day snapshots of the Prometheus Exporters Overview dashboard for the RDS or Aurora host. Here is an example: https://pmmdemo.percona.com/graph/d/vIABtfZmk/prometheus-exporters-overview?refresh=1m&orgId=1&var-interval=$__auto_interval_interval&var-host=rds-aurora1&from=now-7d&to=now

If you could get those over, it might help with understanding the issue. The ‘how to get a snapshot’ is described here: https://www.percona.com/doc/percona-…ting-snapshots

Thanks!

[youngjae7732]

Hello. lorraine.
Thanks for the answer.

Oh. Sorry.
I wanted to see the data quickly, so I eventually deleted old docker and reopened.
So I only have about 12 hours of data.

That’s the snapshot address.
https://snapshots.percona.com/dashboard/snapshot/zGzZjkbIGGHVaIJpnj9gIgyeuTfC00nM

I’ll upload it as an attachement.
The memory problem seems to have disappeared after I reopened pmm docker.
However, as seen in snapshots and attachments, there is sometimes no data in between.
It seems that the data of mysql also can not be collected when there is no data.
Do you know why? I hope for help.

And another question. I can not understand why the following message keeps appearing in the consul.log

2018/05/03 01:56:11 [WARN] consul: skipping reconcile of node {26039e251270 127.0.0.1 8301 map[dc:dc1 vsn:2 role:consul vsn_max:3 raft_vsn:2 build:unknown’’:2c77151 bootstrap:1 wan_join_port:8302 id:db6a705c-aa42-a7f4-6ab0-939a21391956 vsn_min:2 port:8300] alive 1 5 2 2 5 4}

Thank you.

[youngjae7732]

I have confirmed that the current pmm has an error.
Do you know why context deadline exceeded?