# Remove unhealthy targets from Prometheus

**URL:** <https://forums.percona.com/t/remove-unhealthy-targets-from-prometheus/9401>\
**Category:** PMM 2.x\
**Created:** [February 24, 2021, 4:57pm UTC](https://forums.percona.com/t/remove-unhealthy-targets-from-prometheus/9401 "2021-02-24T16:57:09Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![Kevin\_Huet](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/kevin_huet/32/72_2.png) [@Kevin\_Huet](https://forums.percona.com/u/Kevin_Huet)\
**Post date:** [February 24, 2021, 4:57pm UTC](https://forums.percona.com/t/remove-unhealthy-targets-from-prometheus/9401/1 "2021-02-24T16:57:09Z")

</div>

Hi !

I have many agent that has been added in my PMM instance, then the virtual machines has been removed without deleting properly the agents/nodes from the inventory (i.e with `pmm-admin inventory remove command`).

The fact is that, now, I have thousands of unhealthy targets that PMM still wants to scrape every x seconds.

I tried to remove it from the API using the `/v1/inventory/Nodes/Remove` endpoint, but after deleting some of it (apparently successfully according to the empty response body), the number of unhealthy target does not decrease…

Is there something else that I should do ? My server is running low on memory and I’d like to clean this mess as soon as possible.

Any help would be appreciate,

Thanks !

---

<div class="post-metadata">

**Author:** ![Vadim\_Yalovets](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/vadim_yalovets/32/43_2.png) [@Vadim\_Yalovets](https://forums.percona.com/u/Vadim_Yalovets)\
**Post date:** [February 24, 2021, 5:42pm UTC](https://forums.percona.com/t/remove-unhealthy-targets-from-prometheus/9401/2 "2021-02-24T17:42:04Z")

</div>

Hi @Kevin_Huet,

Could you clarify if you can see unhealthy targets in Inventory tables (Services or Nodes)?

> [https://1.2.3.4/graph/d/pmm-inventory/pmm-inventory](https://1.2.3.4/graph/d/pmm-inventory/pmm-inventory)

Is it used the latest PMM2 version?

---

<div class="post-metadata">

**Author:** ![Kevin\_Huet](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/kevin_huet/32/72_2.png) [@Kevin\_Huet](https://forums.percona.com/u/Kevin_Huet)\
**Post date:** [February 25, 2021, 9:13am UTC](https://forums.percona.com/t/remove-unhealthy-targets-from-prometheus/9401/3 "2021-02-25T09:13:36Z")

</div>

Hello,

Yes I see the targets in Inventory tables, both Services and Nodes. And we are currently running version 2.11.1.

---

<div class="post-metadata">

**Author:** ![Kevin\_Huet](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/kevin_huet/32/72_2.png) [@Kevin\_Huet](https://forums.percona.com/u/Kevin_Huet)\
**Post date:** [February 25, 2021, 10:37am UTC](https://forums.percona.com/t/remove-unhealthy-targets-from-prometheus/9401/4 "2021-02-25T10:37:50Z")

</div>

EDIT: I think the “problem” here is that all targets in the `/etc/prometheus.yml` are not updated. Isn’t pmm-managed supposed to do the job ?

---

<div class="post-metadata">

**Author:** ![Vadim\_Yalovets](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/vadim_yalovets/32/43_2.png) [@Vadim\_Yalovets](https://forums.percona.com/u/Vadim_Yalovets)\
**Post date:** [February 25, 2021, 11:22am UTC](https://forums.percona.com/t/remove-unhealthy-targets-from-prometheus/9401/5 "2021-02-25T11:22:44Z")

</div>

> [@Kevin\_Huet](#):
>
> Yes I see the targets in Inventory tables, both Services and Nodes. And we are currently running version 2.11.1.

I have just checked 2.11.1 version and it already has functionality to delete instances on Inventory page directly.  
So you may just remove unnecessary instances on this page.

Please enable force mode in order to remove associated agents.

 ![Screenshot_20210225_131733](https://us1.discourse-cdn.com/flex019/uploads/percona1/original/2X/6/67020c504ef5170b7ed42d047c2e4571417d3019.png)

---

<div class="post-metadata">

**Author:** ![Kevin\_Huet](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/kevin_huet/32/72_2.png) [@Kevin\_Huet](https://forums.percona.com/u/Kevin_Huet)\
**Post date:** [February 25, 2021, 2:18pm UTC](https://forums.percona.com/t/remove-unhealthy-targets-from-prometheus/9401/6 "2021-02-25T14:18:19Z")

</div>

Thanks for your answer, it’s working well now 🥳  
I think on my first try I just removed the hosts from Services but not Nodes, which is far better.

---

<div class="post-metadata">

**Author:** ![nurlan](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/nurlan/32/1523_2.png) [@nurlan](https://forums.percona.com/u/nurlan)\
**Post date:** [March 11, 2021, 8:27pm UTC](https://forums.percona.com/t/remove-unhealthy-targets-from-prometheus/9401/7 "2021-03-11T20:27:07Z")

</div>

Hi @Kevin_Huet,  
Yes, PMM-managed supposed to do that and it was fixed in 2.15.0.
