Not the answer you need?
Register and ask your own question!

pmm-client initContainer fails to start with exitcode 3

Hi guys!
I'm trying to setup PMM based monitoring for perconaservermongodb kubernetes operator.
Server part with percona/pmm-server Helm chart is OK.
But clients are not. I'm setting them with following definition in perconaservermongodbs CR:

pmm:
    enabled: true
    image: percona/percona-server-mongodb-operator:1.4.0-pmm
    serverHost: pmm-service.psmdb
    serverUser: admin

I have also updated "my-cluster-name-secrets" secret to have proper usernames/passwords.
pmm-client initContainer is always failing with exitcode 3:

  - containerID: docker://baf2ea49ad054eed8972cd65d31437ae248b22d8beb55adf00f6ed13d3452db8
    image: percona/percona-server-mongodb-operator:1.4.0-pmm
    imageID: docker-pullable://percona/[email protected]:bf0cdfd9f9971964cb720a92e99da1a75367cf6a07deec9367ca6b80e78b0f89
    lastState:
      terminated:
        containerID: docker://f276c66f0a47d8ce6f626645ffb3255fc03926f961d82de399282972f1af6fd3
        exitCode: 3
        finishedAt: "2020-07-15T11:11:34Z"
        reason: Error
        startedAt: "2020-07-15T11:11:24Z"


Logs are also not so informative:

+ main
+ '[' -z pmm-service.psmdb ']'
+ ARGS+="--server-password=PMM_PASSWORD"
++ sed -e 's/).*//; s/.*(//'
++ grep PING
++ ping -c 1 pmm-service.psmdb
+ PMM_SERVER_IP=172.20.15.45
++ ip route get 172.20.15.45
++ grep 'src '
++ sed -e 's/.* src //; s/ .*//'
+ SRC_ADDR=10.240.90.250
+ CLIENT_NAME=dev-core-rs0-2


At least according to this logs we can see that pmm-server k8s service is resolved properly and reachable.
Then I checked pmm-client docker image entrypoint.sh and I'm pretty sure that error with exitcode 3 happens on that part of main() function:


{ set +x; } 2> /dev/null
    SERVER_RESPONSE_CODE=$(curl -k -s -o /dev/null -w "%{http_code}" "https://${PMM_USER}:$(urlencode ${PMM_PASSWORD})@${PMM_SERVER}/v1/readyz")
    set -x


Because in log we can see succesfull assignment of CLIENT_NAME variable, but pmm2_start() function hasn't started yet, as we dont see echo output from its begining.

The strange thing is that those command works fine when I run it from another shell pod:

[[email protected] /]# echo ${PMM_USER}
admin
[[email protected] /]# echo ${PMM_SERVER}
pmm-service.psmdb
[[email protected] /]# curl -k -s -o /dev/null -w "%{http_code}" "https://${PMM_USER}:$(urlencode ${PMM_PASSWORD})@${PMM_SERVER}/v1/readyz"
200[[email protected] /]# echo $?
0
[[email protected] /]#


I am not sure where to go next... Maybe somebody has faced something similar before or may advice some ideas for further troubleshoot?

Thank you!

Comments

  • TobiasTobias Current User Role Novice
    I'm also struggling to get pmm running with the mongodb operator.
    It manages to register the mongodb in PMM, but then dies after:
    INFO[2020-08-13T06:22:53.513+00:00] time="2020-08-13T06:22:53Z" level=info msg=" - uname" source="node_exporter.go:105"  agentID=/agent_id/854e6754-4ea8-4393-abe7-6604e6a060e6 component=agent-process type=node_exporter
    INFO[2020-08-13T06:22:53.513+00:00] time="2020-08-13T06:22:53Z" level=info msg=" - vmstat" source="node_exporter.go:105"  agentID=/agent_id/854e6754-4ea8-4393-abe7-6604e6a060e6 component=agent-process type=node_exporter
    INFO[2020-08-13T06:22:53.513+00:00] time="2020-08-13T06:22:53Z" level=info msg="HTTP Basic authentication is enabled." source="basic_auth.go:91"  agentID=/agent_id/854e6754-4ea8-4393-abe7-6604e6a060e6 component=agent-process type=node_exporter
    INFO[2020-08-13T06:22:53.513+00:00] time="2020-08-13T06:22:53Z" level=info msg="Starting HTTP server for http://:30100/metrics ..." source="server.go:140"  agentID=/agent_id/854e6754-4ea8-4393-abe7-6604e6a060e6 component=agent-process type=node_exporter
    + pmm-admin add "mongodb" \
                        --skip-connection-check \
                        --server-url="https://xxxxx:[email protected]${PMM_SERVER}/" \
                        --server-insecure-tls \
                        '--uri=mongodb://cluster-monitor:[REDACTED]@127.0.0.1:27017/                     i-c31250bc-d115-4a9e-80f2-8d29600439ec-bla-rs0-2                     :'
    pmm-admin: error: unknown long flag '--uri', try --help
    Yes, the whitespace in the `--uri` line is suspicious, and I don't know where it comes from. Also pmm-admin should have a --uri flag, although not sure if it shuold be inside '-quotes.
    It's all weird.
Sign In or Register to comment.

MySQL, InnoDB, MariaDB and MongoDB are trademarks of their respective owners.
Copyright ©2005 - 2020 Percona LLC. All rights reserved.