Hello everyone
The node information of the aws rds recently created in pmm is not visible.
Some CPU-related information is visible, but memory-related information is not visible.
Below is a portion of rds.log.(pmm-server_2024-08-26_07-11\client\pmm-agent\RDS_EXPORTER pmm-server)
…
time=“2024-08-26T07:10:40.537+00:00” level=error msg=“ts=2024-08-26T07:10:40.535Z caller=scraper.go:89 level=error component=basic metric=ReadIOPS error="InvalidClientTokenId: The security token included in the request is invalid.\n\tstatus code: 403, request id: d46279c3-a7d2-4048-a427-dc9e526e70c4"” agentID=pmm-server/rds component=agent-process type=rds_exporter
time=“2024-08-26T07:10:40.542+00:00” level=error msg=“ts=2024-08-26T07:10:40.540Z caller=scraper.go:89 level=error component=basic metric=BinLogDiskUsage error="InvalidClientTokenId: The security token included in the request is invalid.\n\tstatus code: 403, request id: c71f6330-c771-43a1-8a36-1a82376baa33"” agentID=pmm-server/rds component=agent-process type=rds_exporter
time=“2024-08-26T07:10:55.140+00:00” level=error msg=“ts=2024-08-26T07:10:55.139Z caller=scraper.go:145 level=error component=enhanced msg="Failed to filter log events." error="ResourceNotFoundException: The specified log stream does not exist."” agentID=pmm-server/rds component=agent-process type=rds_exporter
time=“2024-08-26T07:10:55.235+00:00” level=error msg=“ts=2024-08-26T07:10:55.235Z caller=scraper.go:145 level=error component=enhanced msg="Failed to filter log events." error="UnrecognizedClientException: The security token included in the request is invalid."” agentID=pmm-server/rds component=agent-process type=rds_exporter
time=“2024-08-26T07:10:55.674+00:00” level=error msg=“ts=2024-08-26T07:10:55.673Z caller=scraper.go:145 level=error component=enhanced msg="Failed to filter log events." error="ResourceNotFoundException: The specified log stream does not exist."” agentID=pmm-server/rds component=agent-process type=rds_exporter
…
Please let me know if there is a solution
PMMVersion: 2.40.1
rds.log (375.5 KB)
Verify your IAM security token/key have access to the RDS metrics, and CloudWatch.
thank u for your advice~~~
i’ll check out the IAM security token/key and CloudWatch.
Hi everyone.
I checked the information you provided above and confirmed that there were no particular problems. Afterwards, I checked rds.log again and found the following message.
How to solve it
time=“2024-08-29T01:34:05.885+00:00” level=error msg=“ts=2024-08-29T01:34:05.885Z caller=scraper.go:145 level=error component=enhanced msg="Failed to filter log events." error="ResourceNotFoundException: The specified log stream does not exist."” agentID=pmm-server/rds component=agent-process type=rds_exporter
time=“2024-08-29T01:34:06.097+00:00” level=error msg=“ts=2024-08-29T01:34:06.097Z caller=scraper.go:145 level=error component=enhanced msg="Failed to filter log events." error="ResourceNotFoundException: The specified log stream does not exist."” agentID=pmm-server/rds component=agent-process type=rds_exporter
time=“2024-08-29T01:34:27.845+00:00” level=error msg=“ts=2024-08-29T01:34:27.844Z caller=collector.go:78 level=error component=basic msg="No scraper for ap-northeast-2/rds-helpdesk-prd (AKIAT74SYTYET5F27BF2), skipping."” agentID=pmm-server/rds component=agent-process type=rds_exporter
This could be related to [PMM-12426] - Percona JIRA
Please try the below workaround,
As a good workaround here - use AMI roles for the PMM server and rotate them instead of credentials.
Amazon RDS - Percona Monitoring and Management
As you said, I deleted the rds from the inventory list and reconnected it. It worked fine at first, but when I looked at the memory and CPU information as shown in the picture, only the CPU information was displayed.
Also, no information was displayed in the node overview.
I also attached the rds.log and node overview information
rds.log (375.5 KB)
.
Do you have extended metrics enabled for this RDS instance? CloudWatch metrics enabled too? CPU/Disk/Memory information for RDS is pulled from CloudWatch. If these are not enabled, then PMM has nothing to fetch.
extended metrics are enabled for this RDS instance, and CloudWatch metrics are also enabled.
When I deleted it from inventory and reconnected, the memory information came out normally at first, but then it stopped coming out.
Also, this does not appear the same in other rds instances.
Your entire log file is filled with “InvalidClientTokenId: The security token included in the request is invalid.” Please verify your AWS credentials and the policy associated with them.
I uploaded rds.log again today. If you look, there is no InvalidClientTokenId message in rds.log. Thank you for your help.
rds.log (316 KB)
