Hi guys!
I’m trying to setup PMM based monitoring for perconaservermongodb kubernetes operator.
Server part with percona/pmm-server Helm chart is OK.
But clients are not. I’m setting them with following definition in perconaservermongodbs CR:
pmm:
enabled: true
image: percona/percona-server-mongodb-operator:1.4.0-pmm
serverHost: pmm-service.psmdb
serverUser: admin
I have also updated “my-cluster-name-secrets” secret to have proper usernames/passwords.
pmm-client initContainer is always failing with exitcode 3:
- containerID: docker://baf2ea49ad054eed8972cd65d31437ae248b22d8beb55adf00f6ed13d3452db8
image: percona/percona-server-mongodb-operator:1.4.0-pmm
imageID: docker-pullable://percona/percona-server-mongodb-operator@sha256:bf0cdfd9f9971964cb720a92e99da1a75367cf6a07deec9367ca6b80e78b0f89
lastState:
terminated:
containerID: docker://f276c66f0a47d8ce6f626645ffb3255fc03926f961d82de399282972f1af6fd3
exitCode: 3
finishedAt: “2020-07-15T11:11:34Z”
reason: Error
startedAt: “2020-07-15T11:11:24Z”
Logs are also not so informative:
+ main
+ ‘[’ -z pmm-service.psmdb ‘]’
+ ARGS+=“–server-password=PMM_PASSWORD”
++ sed -e ‘s/).//; s/.(//’
++ grep PING
++ ping -c 1 pmm-service.psmdb
+ PMM_SERVER_IP=172.20.15.45
++ ip route get 172.20.15.45
++ grep 'src '
++ sed -e ‘s/.* src //; s/ .*//’
+ SRC_ADDR=10.240.90.250
+ CLIENT_NAME=dev-core-rs0-2
At least according to this logs we can see that pmm-server k8s service is resolved properly and reachable.
Then I checked pmm-client docker image entrypoint.sh and I’m pretty sure that error with exitcode 3 happens on that part of main() function:
{ set +x; } 2> /dev/null
SERVER_RESPONSE_CODE=$(curl -k -s -o /dev/null -w “%{http_code}” “https://${PMM_USER}:$(urlencode ${PMM_PASSWORD})@${PMM_SERVER}/v1/readyz”)
set -x
Because in log we can see succesfull assignment of CLIENT_NAME variable, but pmm2_start() function hasn’t started yet, as we dont see echo output from its begining.
The strange thing is that those command works fine when I run it from another shell pod:
[root@shell /]# echo ${PMM_USER}
admin
[root@shell /]# echo ${PMM_SERVER}
pmm-service.psmdb
[root@shell /]# curl -k -s -o /dev/null -w “%{http_code}” “https://${PMM_USER}:$(urlencode ${PMM_PASSWORD})@${PMM_SERVER}/v1/readyz”
200[root@shell /]# echo $?
0
[root@shell /]#
I am not sure where to go next… Maybe somebody has faced something similar before or may advice some ideas for further troubleshoot?
Thank you!
I’m also struggling to get pmm running with the mongodb operator.
It manages to register the mongodb in PMM, but then dies after:
INFO[2020-08-13T06:22:53.513+00:00] time="2020-08-13T06:22:53Z" level=info msg=" - uname" source="node_exporter.go:105" agentID=/agent_id/854e6754-4ea8-4393-abe7-6604e6a060e6 component=agent-process type=node_exporter
INFO[2020-08-13T06:22:53.513+00:00] time="2020-08-13T06:22:53Z" level=info msg=" - vmstat" source="node_exporter.go:105" agentID=/agent_id/854e6754-4ea8-4393-abe7-6604e6a060e6 component=agent-process type=node_exporter
INFO[2020-08-13T06:22:53.513+00:00] time="2020-08-13T06:22:53Z" level=info msg="HTTP Basic authentication is enabled." source="basic_auth.go:91" agentID=/agent_id/854e6754-4ea8-4393-abe7-6604e6a060e6 component=agent-process type=node_exporter
INFO[2020-08-13T06:22:53.513+00:00] time="2020-08-13T06:22:53Z" level=info msg="Starting HTTP server for http://:30100/metrics ..." source="server.go:140" agentID=/agent_id/854e6754-4ea8-4393-abe7-6604e6a060e6 component=agent-process type=node_exporter
+ pmm-admin add "mongodb" \
--skip-connection-check \
--server-url="https://xxxxx:xxxxx@${PMM_SERVER}/" \
--server-insecure-tls \
'--uri=mongodb://cluster-monitor:[REDACTED]@127.0.0.1:27017/ i-c31250bc-d115-4a9e-80f2-8d29600439ec-bla-rs0-2 :'
pmm-admin: error: unknown long flag '--uri', try --help
Yes, the whitespace in the --uri line is suspicious, and I don’t know where it comes from. Also pmm-admin should have a --uri flag, although not sure if it shuold be inside '-quotes.
It’s all weird.