PMM2 - Password issue after upgrading 2.11 -> 2.25

Percona Monitoring and Management (PMM)
1

:wave: Hi everyone,

Few days ago I did migrate a PMM server from 2.11.1 to 2.25.0, and since the upgrade, when the pmm-agent restart or are upgraded, they loose the connection to the pmm-server, getting the following message:
401; response body="{\"code\":16,\"error\":\"invalid username or password\",\"message\":\"invalid username or password\"}\n

We had the same issue with pmm-agent from 2.6 to 2.18 (didn’t test with a higher version yet).

These credentials didn’t changed, and everybody was correctly connected before. I also get the Invalid username or password error when using the UI…

I tried to create another user with the same password (not working), and then tried to simplify it. It appears that removing the , character inside it make things far better.

I know that it was a big move and I’ve read the CHANGELOG between 2.11.1 and 2.25.0 but haven’t found anything relevant about that.

Are you aware of a similar issue ?

I’m available of course if you need more information.

Thanks a lot :crossed_fingers:

1 Like
2

Hi, do you have anything in logs?
Have you changed admin password since you added pmm-agent?
We introduced push model in PMM 2.12, which requires VM Agent. VM Agent sends metrics to victoria metrics, so it requires actual credentials to do that. Because of multiple requests with invalid credentials grafana blocks user for a while. We are working on it.
If it’s the case then I suggest to update pmm-agent.yml to use actual credentials.

Best regards,
Thanks

1 Like
3

Hi !

No, as I mentionned, the password hasn’t been changed since we added the pmm-agent and everybody was correctly connected before the upgrade :confused:

As a workaround, we’ve created a new user and updated the pmm-agent config but some are still with the “old” creds.

About the logs :
nginx.log

xx.xx.xx.xx - svc_pmm [28/Feb/2022:11:26:29 +0000] "POST /victoriametrics/api/v1/write HTTP/1.1" 401 92 "-" "vmagent" "-"

svc_pmm is the user being invalid since the upgrade (containing the comma inside the password).

grafana.log:

t=2022-02-28T11:30:54+0000 lvl=eror msg="invalid username or password" logger=context error="too many consecutive incorrect login attempts for user - login for user temporarily blocked"

I forgot to mention that some of the pmm-agent were already using version > 2.12 and then when upgrading the server, many of them started to send everything they had in their local buffer (if I understand well).

Are you aware of any issue regarding special character inside the password ?

Thanks a lot for your help :pray:

1 Like
4

Looks like some of your pmm-agent have wrong credentials and because of that grafana blocks your user. That’s why you can’t login via UI and can’t get metrics.

The only workaround that we have now is go through all clients and update credentials.
We are aware of this problem and working on fix.

Regarding special characters we didn’t get any reports about that. I’ll ask our QA team to recheck it.

Thank you.

1 Like