Container pmm-client got error with exit code 1

[wooyoung]

Hi,

I’ve installed Kubernetes Operator for Percona XtraDB Cluster by using the latest helm chart on OpenShift 4.4 setup.

To enable pmm, I did two things:

1. installed pmm-server by using the latest helm chart from https://github.com/Percona-Lab/percona-openshift/tree/master/helm/pmm-server2) change to enable pmm in pxc-db helm chart like below:

values.yaml

<omitted>
pmm:
  enabled: true
  image:
    repository: perconalab/pmm-client
    tag: 1.17.1
  serverHost: monitoring-service
  serverUser: admin

But, during installation, pxc pod got error due to container pmm-client with exit code 1 (see attached):

pod_describe.txt

and container pmm-client log of the pod is (see attached):

pod_log.txt

There is no problem with connectivity between pmm-client and pmm-server including authentication, but it seems pmm-server doesn’t provide /v1/status/leader API which could be seen in above log.

A doubtful point is the difference of installed docker image’s tag between pmm-server and pmm-client:

• pmm-server: percona/pmm-server:2.11.1 (described in https://github.com/Percona-Lab/percona-openshift/blob/master/helm/pmm-server/values.yaml)

• pmm-client: perconalab/pmm-client:1.17.1 (described in https://github.com/percona/percona-helm-charts/blob/master/charts/pxc-db/values.yaml)

could you please let me know what is the root cause of this issue and how to solve it?

Thank you in advance.

Wooyoung

pod_describe.txt (9.63 KB)

pod_log.txt (15.8 KB)

[Evgeniy_Patlan]

hi @wooyoung

Here is the blogpost which show how to install PMM with Kubernetes Operator for Percona XtraDB Cluster

https://www.percona.com/blog/2020/07/23/using-percona-kubernetes-operators-with-percona-monitoring-and-management/

[wooyoung]

Hi @Evgeniy_Patlan

Thanks for your answer. I have two questions:

1. To install pmm server on OpenShift, should we install it with version 1.17.3 only described in the blog you mentioned?

2. Does pmm server with version 2.11.1 provide /v1/status/leader API?

[Evgeniy_Patlan]

hi @wooyoung

For openshift yes it is possible only to use 1.17 PMM for now due to some limitations.

[wooyoung]

Hi @Evgeniy_PatlanI installed 1.17 PMM server and followed guide the blog mentioned.

But I got some error from pmm-client container in pxc pod (below is the container log):

[root@bastion pxc-db]# oc -npercona-test logs -f pxc-db-pxc-0 -c pmm-client
+ '[' -z pxc-monitor-service ']'
+ '[' -n admin ']'
+ ARGS+=' --server-user admin'
+ '[' -n admin ']'
+ ARGS+=' --server-password admin'
++ ping -c 1 pxc-monitor-service
++ grep PING
++ sed -e 's/).*//; s/.*(//'
+ PMM_SERVER_IP=10.21.221.91
++ ip route get 10.21.221.91
++ grep 'src '
++ sed -e 's/.* src //; s/ .*//'
+ SRC_ADDR=10.131.9.34
+ CLIENT_NAME=localhost
+ wait_for_url https://admin:admin@pxc-monitor-service/v1/status/leader 127.0.0.1:8300
+ local URL=https://admin:admin@pxc-monitor-service/v1/status/leader
+ local RESPONSE=127.0.0.1:8300
++ seq 1 60
+ for i in '`seq 1 60`'
+ curl -k https://admin:admin@pxc-monitor-service/v1/status/leader
+ grep 127.0.0.1:8300
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    16  100    16    0     0    141      0 --:--:-- --:--:-- --:--:--   141
+ result=0
+ '[' 0 -eq 0 ']'
+ return
+ pmm-admin config --skip-root --force --server pxc-monitor-service --server-insecure-ssl --bind-address 10.131.9.34 --client-address 10.131.9.34 --client-name localhost --server-user admin --server-password admin
This client is configured with HTTP basic authentication.
However, PMM server is not.


If you forgot to enable password protection on the server, you may want to do so.


Otherwise, run the following command to reset the config and disable authentication:
pmm-admin config --server pxc-monitor-service --server-insecure-ssl

It seems below is the root cause:

+ pmm-admin config --skip-root --force --server pxc-monitor-service --server-insecure-ssl --bind-address 10.131.9.34 --client-address 10.131.9.34 --client-name localhost --server-user admin --server-password admin
This client is configured with HTTP basic authentication.
However, PMM server is not.

Any idea on how to solve this error?

Thank you

[spronin]

Hello @wooyoung ,

starting from PXC Operator 1.7 we added official support for PMM version 2. It works without any hacks. Going forward we will continue the support for PMM v2 in our operators.
PMM server version 2 can be installed in Openshift, but it requires root access.

Please let me know if PMM v2 is an option for you.

[Michael_Coburn]

Just to add to what @spronin mentions, in order to deploy PMM Server 2 in OpenShift you need to apply a Security Context Constraint of anyuid to the project like this example:

oc adm policy add-scc-to-user anyuid -z default -n <project>