Clickhouse FATAL can't find command '/usr/bin/clickhouse-server'

Percona Monitoring and Management (PMM) PMM 2.x
1

Hi Team,

We are trying to upgrade the PMM version from 2.41.1 to 2.43.1, but upgrade is failed with the below errors.

When I check the supervisorctl status clickhouse, then we are getting the below error.

clickhouse FATAL can’t find command ‘/usr/bin/clickhouse-server’

Upgrade log we see the below error:

TASK [clickhouse : Stop and remove clickhouse before update | EL9] *************
failed: [localhost] (item=stop) => {“ansible_loop_var”: “item”, “changed”: true, “cmd”: [“/usr/local/bin/supervisorctl”, “stop”, “clickhouse”], “delta”: “0:00:00.212823”, “end”: “2024-10-23 11:31:49.442841”, “item”: “stop”, “msg”: “non-zero return code”, “rc”: 1, “start”: “2024-10-23 11:31:49.230018”, “stderr”: “”, “stderr_lines”: , “stdout”: “clickhouse: ERROR (no such process)”, “stdout_lines”: [“clickhouse: ERROR (no such process)”]}
failed: [localhost] (item=remove) => {“ansible_loop_var”: “item”, “changed”: true, “cmd”: [“/usr/local/bin/supervisorctl”, “remove”, “clickhouse”], “delta”: “0:00:00.205670”, “end”: “2024-10-23 11:31:49.928712”, “item”: “remove”, “msg”: “non-zero return code”, “rc”: 1, “start”: “2024-10-23 11:31:49.723042”, “stderr”: “”, “stderr_lines”: , “stdout”: “ERROR: no such process/group: clickhouse”, “stdout_lines”: [“ERROR: no such process/group: clickhouse”]}

PLAY RECAP *********************************************************************
localhost : ok=64 changed=12 unreachable=0 failed=1 skipped=73 rescued=0 ignored=0

Jira ticket: [PMM-13444] 2.41.1 to 2.43.1 Upgrade is failed - Percona JIRA

image
image1841×326 43.3 KB

2

Hi @Naresh9999

I’ve performed the upgrade with the following steps:

  • Started docker with PMM-2.41.1
  • Started PS-8.0.39
  • Added the MySQL service to the PMM server
  • Performed upgrade from PMM-2.41.1 to PMM-2.43.1 Using UI button.
  • The upgrade was completed successfully.

Please let us know the exact steps that could be helpful in repeating the behaviour on demand.

3

@Aaditya_Dubey

I have restored the backup to 2.41.1 and again I did an upgrade, but this time I got the below error.

TASK [initialization : Setting current PMM version] ****************************
[WARNING]: Skipping plugin (/usr/lib/python3.9/site-
packages/ansible/plugins/filter/core.py), cannot load: cannot import name
‘_is_unsafe’ from ‘ansible.utils.unsafe_proxy’ (/usr/lib/python3.9/site-
packages/ansible/utils/unsafe_proxy.py)
fatal: [localhost]: FAILED! => {“msg”: “template error while templating string: Could not load "b64decode": "cannot import name ‘_is_unsafe’ from ‘ansible.utils.unsafe_proxy’ (/usr/lib/python3.9/site-packages/ansible/utils/unsafe_proxy.py)". String: {{ current_version_file[‘content’] | b64decode | trim }}. Could not load "b64decode": "cannot import name ‘_is_unsafe’ from ‘ansible.utils.unsafe_proxy’ (/usr/lib/python3.9/site-packages/ansible/utils/unsafe_proxy.py)"”}

PLAY RECAP *********************************************************************
localhost : ok=36 changed=13 unreachable=0 failed=1 skipped=40 rescued=0 ignored=0

4

Hi @Naresh9999

i got a similar warning during upgrades:

TASK [initialization : Setting current PMM version] ****************************
[WARNING]: Skipping plugin (/usr/lib/python3.9/site-
packages/ansible/plugins/filter/core.py), cannot load: cannot import name
‘_is_unsafe’ from ‘ansible.utils.unsafe_proxy’ (/usr/lib/python3.9/site-
packages/ansible/utils/unsafe_proxy.py)
fatal: [localhost]: FAILED! => {“msg”: “template error while templating string: Could not load "b64decode": "cannot import name ‘_is_unsafe’ from ‘ansible.utils.unsafe_proxy’ (/usr/lib/python3.9/site-packages/ansible/utils/unsafe_proxy.py)". String: {{ current_version_file[‘content’] | b64decode | trim }}. Could not load "b64decode": "cannot import name ‘_is_unsafe’ from ‘ansible.utils.unsafe_proxy’ (/usr/lib/python3.9/site-packages/ansible/utils/unsafe_proxy.py)"”}

PLAY RECAP *********************************************************************
localhost : ok=36 changed=13 unreachable=0 failed=1 skipped=40 rescued=0 ignored=0

However, it should not impact your upgrade, and you should have a successful upgrade.

Regarding the initial issue, this issue was only repeated when I removed clickhouse binary from this path /usr/bin/clickhouse inside docker; then I got the following error and the upgrade did not complete and got stuck in an infinite loop:

TASK [clickhouse : Stop and remove clickhouse before update | EL9] *************
failed: [localhost] (item=stop) => {“ansible_loop_var”: “item”, “changed”: true, “cmd”: [“/usr/local/bin/supervisorctl”, “stop”, “clickhouse”], “delta”: “0:00:00.184625”, “end”: “2024-10-24 14:13:26.703853”, “item”: “stop”, “msg”: “non-zero return code”, “rc”: 1, “start”: “2024-10-24 14:13:26.519228”, “stderr”: “”, “stderr_lines”: , “stdout”: “clickhouse: ERROR (no such process)”, “stdout_lines”: [“clickhouse: ERROR (no such process)”]}
failed: [localhost] (item=remove) => {“ansible_loop_var”: “item”, “changed”: true, “cmd”: [“/usr/local/bin/supervisorctl”, “remove”, “clickhouse”], “delta”: “0:00:00.164045”, “end”: “2024-10-24 14:13:27.071542”, “item”: “remove”, “msg”: “non-zero return code”, “rc”: 1, “start”: “2024-10-24 14:13:26.907497”, “stderr”: “”, “stderr_lines”: , “stdout”: “ERROR: no such process/group: clickhouse”, “stdout_lines”: [“ERROR: no such process/group: clickhouse”]}

Can you check if the clickhouse binary was present inside the Docker or PMM host machine before you upgraded?

5

Hi, @Aaditya_Dubey.

Thanks for all of the testing; we are currently in production; therefore, we restored the prior backup and upgraded the PMM server, which resulted in the ansible utils and Python warnings.

Before restoring the backup, Nurlan suggested installing and testing clickhouse-server-23.8.2.7, i have installed it; however, it did not start after that, even when we restarted the Docker service. So no luck, and our management instructed us to restore the backup and attempt the upgrade.

1 Like