Remove unhealthy targets from Prometheus

Percona Monitoring and Management (PMM) PMM 2.x
1

Hi !

I have many agent that has been added in my PMM instance, then the virtual machines has been removed without deleting properly the agents/nodes from the inventory (i.e with pmm-admin inventory remove command).

The fact is that, now, I have thousands of unhealthy targets that PMM still wants to scrape every x seconds.

I tried to remove it from the API using the /v1/inventory/Nodes/Remove endpoint, but after deleting some of it (apparently successfully according to the empty response body), the number of unhealthy target does not decrease…

Is there something else that I should do ? My server is running low on memory and I’d like to clean this mess as soon as possible.

Any help would be appreciate,

Thanks !

2

Hi @Kevin_Huet,

Could you clarify if you can see unhealthy targets in Inventory tables (Services or Nodes)?

https://1.2.3.4/graph/d/pmm-inventory/pmm-inventory

Is it used the latest PMM2 version?

1 Like
3

Hello,

Yes I see the targets in Inventory tables, both Services and Nodes. And we are currently running version 2.11.1.

1 Like
4

EDIT: I think the “problem” here is that all targets in the /etc/prometheus.yml are not updated. Isn’t pmm-managed supposed to do the job ?

1 Like
5

I have just checked 2.11.1 version and it already has functionality to delete instances on Inventory page directly.
So you may just remove unnecessary instances on this page.

Please enable force mode in order to remove associated agents.

Screenshot_20210225_131733
Screenshot_20210225_1317331817Ă—556 67 KB

1 Like
6

Thanks for your answer, it’s working well now :partying_face:
I think on my first try I just removed the hosts from Services but not Nodes, which is far better.

1 Like
7

Hi @Kevin_Huet,
Yes, PMM-managed supposed to do that and it was fixed in 2.15.0.

1 Like