Problem with PMM - Exit code 70

Percona Monitoring and Management (PMM) PMM 3.x
1

Hello, I have a very weird problem with PMM. The Clickhouse inside restarts continuously with exit code 70. It is deployed in Kubernetes from Helm Chart version 1.4.0 and App version 3.0.0

The log messages I see are like:

2025-05-21 15:03:46,213 INFO spawned: 'clickhouse' with pid 656566
2025-05-21 15:03:47,395 INFO success: clickhouse entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-05-21 15:03:47,463 INFO exited: clickhouse (exit status 70; not expected)

Please, advise on what could be wrong.
Thank you in advance.

2

Hi @Alexander_Tachev,

You can find the actual ClickHouse logs under (inside the pmm server pod):

/srv/logs/

You can also get logs via:

https://address-of-your-pmm-server/logs.zip
1 Like
3

Huge thanks for your help. The error is:

2025.05.22 06:44:56.559067 [ 1245807 ] {} <Error> Application: filesystem error: in remove_all: Directory not empty ["/srv/clickhouse/store/34a/34a44088-121c-4d8d-846b-5c1ce49e80cd/delete_tmp_202502_10543_10543_0/"]

The folders are owned by root and the pmm user is not allowed to remove them. Is there any other command I can execute with it and tell ClickHouse to remove those folders?

4

Did you upgrade from PMM 2? In this case please follow this instruction to change ownership before upgrade Migrate PMM 2 to PMM 3 - Percona Monitoring and Management

1 Like
5

Sincere thanks to both of you :heart:

1 Like