pmm-client initContainer is failing with exit code 3

Percona Monitoring and Management (PMM) PMM 2.x
1

Hi guys!I’m trying to setup PMM based monitoring for perconaservermongodb kubernetes operator.Server part with percona/pmm-server Helm chart is OK.But clients are not. I’m setting them with following definition in perconaservermongodbs CR:

pmm:
enabled: true
image: percona/percona-server-mongodb-operator:1.4.0-pmm
serverHost: pmm-service.psmdb
serverUser: admin

I have also updated “my-cluster-name-secrets” secret to have proper usernames/passwords.pmm-client initContainer is always failing with exitcode 3:
- containerID: docker://baf2ea49ad054eed8972cd65d31437ae248b22d8beb55adf00f6ed13d3452db8
  image: percona/percona-server-mongodb-operator:1.4.0-pmm
  imageID: docker-pullable://percona/percona-server-mongodb-operator@sha256:bf0cdfd9f9971964cb720a92e99da1a75367cf6a07deec9367ca6b80e78b0f89
  lastState:
    terminated:
      containerID: docker://f276c66f0a47d8ce6f626645ffb3255fc03926f961d82de399282972f1af6fd3
      exitCode: 3
      finishedAt: “2020-07-15T11:11:34Z”
      reason: Error
      startedAt: “2020-07-15T11:11:24Z”

Logs are also not so informative:

+ main<br>+ '[' -z pmm-service.psmdb ']'<br>+ ARGS+="--server-password=PMM_PASSWORD"<br>++ sed -e 's/).*//; s/.*(//'<br>++ grep PING<br>++ ping -c 1 pmm-service.psmdb<br>+ PMM_SERVER_IP=172.20.15.45<br>++ ip route get 172.20.15.45<br>++ grep 'src '<br>++ sed -e 's/.* src //; s/ .*//'<br>+ SRC_ADDR=10.240.90.250<br>+ CLIENT_NAME=dev-core-rs0-2<br>


At least according to this logs we can see that pmm-server k8s service is resolved properly and reachable.Then I checked pmm-client docker image entrypoint.sh and I’m pretty sure that error with exitcode 3 happens on that part of main() function:

<br>{ set +x; } 2&gt; /dev/null<br>&nbsp;&nbsp;&nbsp; SERVER_RESPONSE_CODE=$(curl -k -s -o /dev/null -w "%{http_code}" "https://${PMM_USER}:$(urlencode ${PMM_PASSWORD})@${PMM_SERVER}/v1/readyz")<br>&nbsp;&nbsp;&nbsp; set -x


Because in log we can see succesfull assignment of CLIENT_NAME variable, but pmm2_start() function hasn’t started yet, as we dont see echo output from its begining.
The strange thing is that those command works fine when I run it from another shell pod:

[root@shell /]# echo ${PMM_USER}<br>admin<br>[root@shell /]# echo ${PMM_SERVER}<br>pmm-service.psmdb<br>[root@shell /]# curl -k -s -o /dev/null -w "%{http_code}" "https://${PMM_USER}:$(urlencode ${PMM_PASSWORD})@${PMM_SERVER}/v1/readyz"<br>200[root@shell /]# echo $?<br>0<br>[root@shell /]#


I am not sure where to go next… Maybe somebody has faced something similar before or may advice some ideas for further troubleshoot?
Thank you!






2

Logs were not formatted properly here, please see my question here:pmm-client initContainer fails to start with exitcode 3 - PMM 2.x - Percona Community Forum