PMM Client Upgrade Issues 2.x to 3.x

Percona Monitoring and Management (PMM)
1

Description:

I have recently upgraded our kubernetes mongodb deployment to use PMM 3.8.1. I have 4 deployments in our NP environment that all talk back to the same PMM install. The PMM migration went well. When I went to upgrade the PMM client for the first mongodb deployment, the pmm-client kept crashing.

Upon further investigation I found that the operator appears to be injecting the wrong values into the stateful sets. Specifically PMM_AGENT_CONFIG_FILE is being pushed as /usr/local/percona/pmm2/config/pmm-agent.yaml instead of /usr/local/percona/pmm/config/pmm-agent.yaml.

The helm values for the mongodb deployment inject PMM_API_KEY, PMM_ADMIN_USER and PMM_ADMIN_PASSWORD as part of the terragrunt build. I have read that because of support for pmm2 and pmm3 together this may be what is causing the issue. Looking through the operator code there is this piece.

secret.Data[api.PMMServerToken]

However, I do not see that as an option in the helm value examples for mongo at percona-helm-charts/charts/psmdb-db/values.yaml at main · percona/percona-helm-charts · GitHub .

If I edit the stateful set and change the PMM_AGENT_CONFIG_FILE to /usr/local/percona/pmm/config/pmm-agent.yaml and scale up the set, it connects correctly.

I am sure I am missing something simple but can’t seem to find docs around how exactly to get the new pmm-agents to use API tokens instead of Keys which I am assuming is why the operator is thinking PMM2 when I really want PMM3 and is then injecting the pmm2 path into the stateful sets PMM_AGENT_CONFIG_FILE variable.

Operator

Chart: psmdb-operator-1.21.3

APP VERSION: 1.21.2

PSMDB

crVersion: 1.21.2

Chart: 1.21.2

pmm-agent: 3.5 but tried a few

PMM

Chart 1.8.1

Any thoughts or advice would be greatly appreciated.

Thank you,

2

Ok, I need to do some testing but think I figured it out. If someone can just confirm it would be great. It looks like if the value of PMM_SERVER_API_KEY is an api key, then it assumes pmm2, if it is an API token it assumes pmm3.

I tested by manually updating the secret to a new token, let the operator recycle the containers and then ran kubectl get sts
-n -o yaml | grep -A5 -B5 PMM_AGENT_CONFIG_FILE, this resulted in;

name: PMM_AGENT_CONFIG_FILE
value: /usr/local/percona/pmm/config/pmm-agent.yaml

and not

name: PMM_AGENT_CONFIG_FILE
value: /usr/local/percona/pmm2/config/pmm-agent.yaml

3

So did another environment and think that did not work. It did not automatically change from pmm2 to pmm. Is there something that needs to be passed to ensure the use of pmm3? I don’t like the idea of needing to edit the stateful sets to change PMM_AGENT_CONFIG_FILE from /usr/local/percona/pmm2/config/pmm-agent.yaml to /usr/local/percona/pmm/config/pmm-agent.yaml.

4

Ok, pretty sure I got it now. use PMM_SERVER_TOKEN instead of PMM_SERVER_API_KEY. Am I just completely missing something in the docs or is this not well documented?

5

Right, PMM 3 will use service account token

Hope that helps