Monitoring not working with XtraDB Cluster

[Shashankft9]

Hey, can someone help me with setting up monitoring for the xtradb cluster?
I am following the documentation for using PMM: https://www.percona.com/blog/2020/07/23/using-percona-kubernetes-operators-with-percona-monitoring-and-management/

I have deployed the server using: helm install monitoring percona/pmm-server --set platform=kubernetes --version 2.7.0 --set “credentials.password=supa|^|pazz”

• the server is deployed correctly.
• but when I enable pmm in deploy/cr.yaml, all the pxc pods come up with 1/2 ready state, getting this error in pmm-client:
  Invalid PMM Server URL "https://admin:supa||pazz@monitoring-service/": parse [https://admin:supa](https://admin:supa/)||pazz@monitoring-service/: net/url: invalid userinfo.
  OR
  SRC_ADDR=x.x.x.x
  • CLIENT_NAME=cluster1-pxc-1
    ++ curl -k -s -o /dev/null -w ‘%{http_code}’ ‘https://admin:supa|^|pazz@monitoring-service/v1/readyz’
  • SERVER_RESPONSE_CODE=000
• I do have the monitoring-service in pxc namespace, and I can even curl to it:
  curl https://<monitoring-service-ip>/v1/readyz -k -vvv, it gives me a 200 response.

one question:
Is there any way, I could simply get the prometheus metrics, and instead of using the PMM, use my own grafana instance?

[vadimtk]

We do not expose prometheus metrics.
As a recommendation you may try our new recently releases PXC Operator 1.7.0, which has full support for PMM 2.0. It should be much easier to get it running.

[Shashankft9]

I just removed the 1.6 version, and started fresh with the v1.7
I deploy the PMM server using this:
helm install monitoring percona/pmm-server --set platform=kubernetes --version 2.7.0 --set "credentials.password=supa|^|pazz"
I then manually change the password through GUI to “newpass”.
Now while installing the xtradb cluster, I am changing the secret to look like this:
apiVersion: v1
kind: Secret
metadata:
name: my-cluster-secrets
type: Opaque
stringData:
root: root_password
xtrabackup: backup_password
monitor: monitory
clustercheck: clustercheckpassword
proxyadmin: admin_password
pmmserver: newpass
operator: operatoradmin
.
I changed the cr.yaml to have the serverUser as “admin”. I create the cr, and I am still getting errors in the pmm-client:
Checking local pmm-agent status…
pmm-agent is not running.
Registering pmm-agent on PMM Server…
Failed to register pmm-agent on PMM Server: Invalid username or password.
Please check username and password.
pmm-agent setup exited with 1.

Note that, I can login with admin/newpass as credentials on the pmm server.

[Shashankft9]

and when I switch the image of the pmm server statefulset to percona/pmm-server:2.0 I get following error:
pmm-agent is not running.
Registering pmm-agent on PMM Server…
Failed to register pmm-agent on PMM Server: Internal server error…
pmm-agent setup exited with 1.

Can you please try to reproduce on a kubernetes cluster.