Hi guys!I’m trying to setup PMM based monitoring for perconaservermongodb kubernetes operator.Server part with percona/pmm-server Helm chart is OK.But clients are not. I’m setting them with following definition in perconaservermongodbs CR:
pmm:
enabled: true
image: percona/percona-server-mongodb-operator:1.4.0-pmm
serverHost: pmm-service.psmdb
serverUser: admin
I have also updated “my-cluster-name-secrets” secret to have proper usernames/passwords.pmm-client initContainer is always failing with exitcode 3:
- containerID: docker://baf2ea49ad054eed8972cd65d31437ae248b22d8beb55adf00f6ed13d3452db8
image: percona/percona-server-mongodb-operator:1.4.0-pmm
imageID: docker-pullable://percona/percona-server-mongodb-operator@sha256:bf0cdfd9f9971964cb720a92e99da1a75367cf6a07deec9367ca6b80e78b0f89
lastState:
terminated:
containerID: docker://f276c66f0a47d8ce6f626645ffb3255fc03926f961d82de399282972f1af6fd3
exitCode: 3
finishedAt: “2020-07-15T11:11:34Z”
reason: Error
startedAt: “2020-07-15T11:11:24Z”
Logs are also not so informative:
+ main<br>+ '[' -z pmm-service.psmdb ']'<br>+ ARGS+="--server-password=PMM_PASSWORD"<br>++ sed -e 's/).*//; s/.*(//'<br>++ grep PING<br>++ ping -c 1 pmm-service.psmdb<br>+ PMM_SERVER_IP=172.20.15.45<br>++ ip route get 172.20.15.45<br>++ grep 'src '<br>++ sed -e 's/.* src //; s/ .*//'<br>+ SRC_ADDR=10.240.90.250<br>+ CLIENT_NAME=dev-core-rs0-2<br>
At least according to this logs we can see that pmm-server k8s service is resolved properly and reachable.Then I checked pmm-client docker image entrypoint.sh and I’m pretty sure that error with exitcode 3 happens on that part of main() function:
<br>{ set +x; } 2> /dev/null<br> SERVER_RESPONSE_CODE=$(curl -k -s -o /dev/null -w "%{http_code}" "https://${PMM_USER}:$(urlencode ${PMM_PASSWORD})@${PMM_SERVER}/v1/readyz")<br> set -x
Because in log we can see succesfull assignment of CLIENT_NAME variable, but pmm2_start() function hasn’t started yet, as we dont see echo output from its begining.
The strange thing is that those command works fine when I run it from another shell pod:
[root@shell /]# echo ${PMM_USER}<br>admin<br>[root@shell /]# echo ${PMM_SERVER}<br>pmm-service.psmdb<br>[root@shell /]# curl -k -s -o /dev/null -w "%{http_code}" "https://${PMM_USER}:$(urlencode ${PMM_PASSWORD})@${PMM_SERVER}/v1/readyz"<br>200[root@shell /]# echo $?<br>0<br>[root@shell /]#
I am not sure where to go next… Maybe somebody has faced something similar before or may advice some ideas for further troubleshoot?
Thank you!