Can't change passwords in my-cluster-secrets

MySQL & MariaDB Percona Operator for MySQL
1

Hello, I’m facing a problem with passwords changing.
First I deploy percona-operator and my-cluster-secrets with default passwords:

root: cm9vdF9wYXNzd29yZA==
xtrabackup: YmFja3VwX3Bhc3N3b3Jk
monitor: bW9uaXRvcg==
clustercheck: Y2x1c3RlcmNoZWNrcGFzc3dvcmQ=
proxyadmin: YWRtaW5fcGFzc3dvcmQ=
pmmserver: c3VwYXxefHBheno=
operator: b3BlcmF0b3JhZG1pbg==
and everything’s ok.

Then I change secrets to custom this way

stringData:
root: f13a826771
operator: xkijmtfjz8
monitor: 3i2sc5ehy2
clustercheck: x1o5xeol9e
proxyadmin: 1d567w40gs
xtrabackup: f894n8na58
pmmserver: x8gvn4n6kr

and remove these rows in proxysql config:

  admin_variables =
  {
    admin_credentials="proxyadmin:admin_password"
    cluster_username="proxyadmin"
    cluster_password="admin_password"

  mysql_variables=
  {
    monitor_password="monitor"

and proxysql pods restarting and start giving an errors:

2021-01-20 13:43:07 MySQL_Session.cpp:4690:handler___status_CONNECTING_CLIENT___STATE_SERVER_HANDSHAKE(): [ERROR] ProxySQL Error: Access denied for user ‘proxyadmin’@‘127.0.0.1’ (using password: YES)
2021-01-20 13:43:07 MySQL_Monitor.cpp:1785:monitor_galera_thread(): [ERROR] Server percona-pxc-1.percona-pxc.***.svc.cluster.local:3306 missed 3 Galera checks. Assuming offline

So,

  1. what could be the reason?
  2. what’s the correct procedure of password changing?
  3. should these options exist: “admin_credentials”, “cluster_username”, “cluster_password”, “monitor_password” in proxysql configuration?
  4. what is the purpose of the internal-percona secret, should there be any actions with it?

Can provide cr.yml and operator-deployment.yml manifests but can’t attach it for now because it restricted for the new users.

Thank you.

2

Are you sure your passwords are base64 encoded in secrets.yaml? And next, did you re-deploy the secrets file?

3

Yes, k8s codes them from string to base64 by itself if there is “stringData” option. They match with manual coding by “echo -n … | base64”

First it was k apply -f cluster-secrets-default.yml with default passwords, then k apply -f cluster-secrets-new.yml with custom.

I just tried setting up a cluster with initially custom passwords in secret. Errors with proxyadmin persist. But there is a strange thing: I can connect with the custom root password directly to mysql from pxc pod. Seems that the secret applies, but not for all passwords.

4

Hi @dmitriiu

is this bug relevant to your case [K8SPXC-641] Update of secret for proxyadmin user does not work properly - Percona JIRA ?

5

hi, I didn’t notice that the new password was successfully applied to only one of the three nodes, but otherwise the bug is similar.

6

Yes, bug K8SPXC-641 covers this issue. It will be shipped in release 1.8.0.