# Clickhouse FATAL can't find command '/usr/bin/clickhouse-server'
**URL:** https://forums.percona.com/t/clickhouse-fatal-cant-find-command-usr-bin-clickhouse-server/34560
**Category:** PMM 2.x
**Created:** [October 23, 2024, 12:14pm UTC](https://forums.percona.com/t/clickhouse-fatal-cant-find-command-usr-bin-clickhouse-server/34560 "2024-10-23T12:14:32Z")
**Posts on this page:** 5
**Page:** 1
### Author:  [@Naresh9999](https://forums.percona.com/u/Naresh9999)
#### Post date: [October 23, 2024, 12:14pm UTC](https://forums.percona.com/t/clickhouse-fatal-cant-find-command-usr-bin-clickhouse-server/34560/1 "2024-10-23T12:14:32Z")
Hi Team,
We are trying to upgrade the PMM version from 2.41.1 to 2.43.1, but upgrade is failed with the below errors.
When I check the supervisorctl status clickhouse, then we are getting the below error.
clickhouse FATAL can’t find command ‘/usr/bin/clickhouse-server’
Upgrade log we see the below error:
TASK [clickhouse : Stop and remove clickhouse before update | EL9] \*\*\*\*\*\*\*\*\*\*\*\*\*
failed: [localhost] (item=stop) =\> {“ansible\_loop\_var”: “item”, “changed”: true, “cmd”: [“/usr/local/bin/supervisorctl”, “stop”, “clickhouse”], “delta”: “0:00:00.212823”, “end”: “2024-10-23 11:31:49.442841”, “item”: “stop”, “msg”: “non-zero return code”, “rc”: 1, “start”: “2024-10-23 11:31:49.230018”, “stderr”: “”, “stderr\_lines”: , “stdout”: “clickhouse: ERROR (no such process)”, “stdout\_lines”: [“clickhouse: ERROR (no such process)”]}
failed: [localhost] (item=remove) =\> {“ansible\_loop\_var”: “item”, “changed”: true, “cmd”: [“/usr/local/bin/supervisorctl”, “remove”, “clickhouse”], “delta”: “0:00:00.205670”, “end”: “2024-10-23 11:31:49.928712”, “item”: “remove”, “msg”: “non-zero return code”, “rc”: 1, “start”: “2024-10-23 11:31:49.723042”, “stderr”: “”, “stderr\_lines”: , “stdout”: “ERROR: no such process/group: clickhouse”, “stdout\_lines”: [“ERROR: no such process/group: clickhouse”]}
PLAY RECAP \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*
localhost : ok=64 changed=12 unreachable=0 failed=1 skipped=73 rescued=0 ignored=0
Jira ticket: [[PMM-13444] 2.41.1 to 2.43.1 Upgrade is failed - Percona JIRA](https://perconadev.atlassian.net/browse/PMM-13444)
---
### Author:  [@Aaditya\_Dubey](https://forums.percona.com/u/Aaditya_Dubey)
#### Post date: [October 24, 2024, 9:07am UTC](https://forums.percona.com/t/clickhouse-fatal-cant-find-command-usr-bin-clickhouse-server/34560/2 "2024-10-24T09:07:14Z")
Hi @Naresh9999
I’ve performed the upgrade with the following steps:
- Started docker with PMM-2.41.1
- Started PS-8.0.39
- Added the MySQL service to the PMM server
- Performed upgrade from PMM-2.41.1 to PMM-2.43.1 Using UI button.
- The upgrade was completed successfully.
Please let us know the exact steps that could be helpful in repeating the behaviour on demand.
---
### Author:  [@Naresh9999](https://forums.percona.com/u/Naresh9999)
#### Post date: [October 24, 2024, 12:46pm UTC](https://forums.percona.com/t/clickhouse-fatal-cant-find-command-usr-bin-clickhouse-server/34560/3 "2024-10-24T12:46:26Z")
@Aaditya_Dubey
I have restored the backup to 2.41.1 and again I did an upgrade, but this time I got the below error.
TASK [initialization : Setting current PMM version] \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*
[WARNING]: Skipping plugin (/usr/lib/python3.9/site-
packages/ansible/plugins/filter/core.py), cannot load: cannot import name
‘\_is\_unsafe’ from ‘ansible.utils.unsafe\_proxy’ (/usr/lib/python3.9/site-
packages/ansible/utils/unsafe\_proxy.py)
fatal: [localhost]: FAILED! =\> {“msg”: “template error while templating string: Could not load "b64decode": "cannot import name ‘\_is\_unsafe’ from ‘ansible.utils.unsafe\_proxy’ (/usr/lib/python3.9/site-packages/ansible/utils/unsafe\_proxy.py)". String: {{ current\_version\_file[‘content’] | b64decode | trim }}. Could not load "b64decode": "cannot import name ‘\_is\_unsafe’ from ‘ansible.utils.unsafe\_proxy’ (/usr/lib/python3.9/site-packages/ansible/utils/unsafe\_proxy.py)"”}
PLAY RECAP \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*
localhost : ok=36 changed=13 unreachable=0 failed=1 skipped=40 rescued=0 ignored=0
---
### Author:  [@Aaditya\_Dubey](https://forums.percona.com/u/Aaditya_Dubey)
#### Post date: [October 24, 2024, 3:23pm UTC](https://forums.percona.com/t/clickhouse-fatal-cant-find-command-usr-bin-clickhouse-server/34560/4 "2024-10-24T15:23:20Z")
Hi @Naresh9999
i got a similar warning during upgrades:
> TASK [initialization : Setting current PMM version] \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*
> [WARNING]: Skipping plugin (/usr/lib/python3.9/site-
> packages/ansible/plugins/filter/core.py), cannot load: cannot import name
> ‘\_is\_unsafe’ from ‘ansible.utils.unsafe\_proxy’ (/usr/lib/python3.9/site-
> packages/ansible/utils/unsafe\_proxy.py)
> fatal: [localhost]: FAILED! =\> {“msg”: “template error while templating string: Could not load "b64decode": "cannot import name ‘\_is\_unsafe’ from ‘ansible.utils.unsafe\_proxy’ (/usr/lib/python3.9/site-packages/ansible/utils/unsafe\_proxy.py)". String: {{ current\_version\_file[‘content’] | b64decode | trim }}. Could not load "b64decode": "cannot import name ‘\_is\_unsafe’ from ‘ansible.utils.unsafe\_proxy’ (/usr/lib/python3.9/site-packages/ansible/utils/unsafe\_proxy.py)"”}
>
> PLAY RECAP \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*
> localhost : ok=36 changed=13 unreachable=0 failed=1 skipped=40 rescued=0 ignored=0
However, it should not impact your upgrade, and you should have a successful upgrade.
Regarding the initial issue, this issue was only repeated when I removed `clickhouse` binary from this path `/usr/bin/clickhouse` inside docker; then I got the following error and the upgrade did not complete and got stuck in an infinite loop:
> TASK [clickhouse : Stop and remove clickhouse before update | EL9] \*\*\*\*\*\*\*\*\*\*\*\*\*
> failed: [localhost] (item=stop) =\> {“ansible\_loop\_var”: “item”, “changed”: true, “cmd”: [“/usr/local/bin/supervisorctl”, “stop”, “clickhouse”], “delta”: “0:00:00.184625”, “end”: “2024-10-24 14:13:26.703853”, “item”: “stop”, “msg”: “non-zero return code”, “rc”: 1, “start”: “2024-10-24 14:13:26.519228”, “stderr”: “”, “stderr\_lines”: , “stdout”: “clickhouse: ERROR (no such process)”, “stdout\_lines”: [“clickhouse: ERROR (no such process)”]}
> failed: [localhost] (item=remove) =\> {“ansible\_loop\_var”: “item”, “changed”: true, “cmd”: [“/usr/local/bin/supervisorctl”, “remove”, “clickhouse”], “delta”: “0:00:00.164045”, “end”: “2024-10-24 14:13:27.071542”, “item”: “remove”, “msg”: “non-zero return code”, “rc”: 1, “start”: “2024-10-24 14:13:26.907497”, “stderr”: “”, “stderr\_lines”: , “stdout”: “ERROR: no such process/group: clickhouse”, “stdout\_lines”: [“ERROR: no such process/group: clickhouse”]}
Can you check if the `clickhouse` binary was present inside the Docker or PMM host machine before you upgraded?
---
### Author:  [@Naresh9999](https://forums.percona.com/u/Naresh9999)
#### Post date: [October 24, 2024, 4:10pm UTC](https://forums.percona.com/t/clickhouse-fatal-cant-find-command-usr-bin-clickhouse-server/34560/5 "2024-10-24T16:10:44Z")
Hi, @Aaditya_Dubey.
Thanks for all of the testing; we are currently in production; therefore, we restored the prior backup and upgraded the PMM server, which resulted in the ansible utils and Python warnings.
Before restoring the backup, Nurlan suggested installing and testing clickhouse-server-23.8.2.7, i have installed it; however, it did not start after that, even when we restarted the Docker service. So no luck, and our management instructed us to restore the backup and attempt the upgrade.