With the most recent tech preview of PMM on Kubernetes I was going through this blog
I am able to setup pmm and its working fine, next I want to get mysql installed onto the cluster and have its stats going to the pmm so that I can make use of the fancy dashboards.
I tried below command
helm install my-db --set pmm.enabled=true --set pmm.image="percona/pmm-client:2.28.0" --set pmm.serverHost="monitoring-service" percona/pxc-db
IT gives the below error
Error: INSTALLATION FAILED: template: pxc-db/templates/cluster.yaml:431:18: executing "pxc-db/templates/cluster.yaml" at <$pmm.image.repository>: can't evaluate field repository in type interface {}
I did check the templates cluster.yml for mysql - it has image: repo and tag both.
Anything I am doing wrong while installing this ?
2 Likes
Hi @Saiyam_Pathak !
Seems to me you will need to split the pmm.image into pmm.image.repository and pmm.image.tag variables so you could try like this:
helm install my-db --set pmm.enabled=true --set pmm.image.repository="percona/pmm-client" --set pmm.image.tag="2.28.0" --set pmm.serverHost="monitoring-service" percona/pxc-db
2 Likes
Hi @Tomislav_Plavcic thank you for your response
The issue I am facing is I run the pmm instlal on Kubernetes
this is fine
Then I try to install mysql cluster using
here after
helm install my-op percona/pxc-operator
I run below
helm install my-db --set pmm.enabled=true --set pmm.image.repository="percona/pmm-client" --set pmm.image.tag="2.28.0" --set pmm.serverHost="monitoring-service" percona/pxc-db
as per the docs we just need three things to get the metrics flowing to PMM on the same cluster (Percona Monitoring and Management in Kubernetes is now in Tech Preview - Percona Database Performance Blog)
Here is what I have on the cluster and I am not able to get the metrics flowing to pmm MQSQL
as I am getting Failed to register pmm-agent on PMM Server: invalid username or password
I have generated the key from the pmm dashboard, how can I add that now ?
1 Like
Hi @Saiyam_Pathak !
At the current moment it is possible, but only if you create your secret with cluster passwords before starting the pxc cluster with helm and while starting you need to specify pxc.clusterSecretName with the name of your secret which includes the pmmserverkey string.
So how it might look is, you create a secret which can look something like this:
$ cat secrets.yaml
apiVersion: v1
kind: Secret
metadata:
name: my-cluster-secrets
type: Opaque
stringData:
root: root_password
xtrabackup: backup_password
monitor: monitory
clustercheck: clustercheckpassword
proxyadmin: admin_password
pmmserverkey: api_key_value
operator: operatoradmin
replication: repl_password
then you start PXC with something like this (notice --set pxc.clusterSecretName=“my-cluster-secrets”):
helm install my-db --set pmm.enabled=true --set pmm.image.repository="percona/pmm-client" --set pmm.image.tag="2.28.0" --set pmm.serverHost="monitoring-service" --set pxc.clusterSecretName="my-cluster-secrets" percona/pxc-db
I have opened a Jira ticket to support pmmserverkey without the need to first create a custom secret (so you may track the progress): [K8SPXC-1054] Helm - enable support for pmmserverkey in the secret - Percona JIRA
1 Like
Perfect, Let me try this and will report back.
1 Like
Hi @Saiyam_Pathak !
The fix for K8SPXC-1054 ticket was merged so it means you can now specify your PMM key directly in the command line or values file.
Something like this should work: --set secrets.passwords.pmmserverkey="..."
Please check if it works and report back.
Thanks!
1 Like
Hi @Tomislav_Plavcic I tried but I am facing weird issue
BElow are my steps - I also tried by creating secrets file on a separate cluster, that also didnt work
helm install pmm --set service.type="LoadBalancer" percona/pmm
helm install my-op percona/pxc-operator
export ADMIN_PASS=$(kubectl get secret pmm-secret --namespace default -o jsonpath='{.data.PMM_ADMIN_PASSWORD}' | base64 --decode)
echo $ADMIN_PASS
tU4pv6~(!f!g5WU=
helm install my-db --set pmm.enabled=true --set pmm.image.repository="percona/pmm-client" --set pmm.image.tag="2.28.0" --set pmm.serverHost="monitoring-service" --set pmm.secrets.passwords.pmmserverkey="tU4pv6~(5WU=" percona/pxc-db
kubectl get pods
NAME READY STATUS RESTARTS AGE
my-op-pxc-operator-7fb5b88658-mrd87 1/1 Running 0 7m59s
pmm-0 1/1 Running 0 9m25s
my-db-pxc-db-pxc-0 4/4 Running 0 5m19s
my-db-pxc-db-haproxy-0 3/3 Running 0 5m19s
my-db-pxc-db-haproxy-1 3/3 Running 0 3m44s
my-db-pxc-db-haproxy-2 3/3 Running 0 3m18s
my-db-pxc-db-pxc-1 4/4 Running 0 3m49s
my-db-pxc-db-pxc-2 4/4 Running 0 2m19s
mm-agent is running.
Registering pmm-agent on PMM Server...
Failed to register pmm-agent on PMM Server: invalid username or password
Please check username and password.
[u'pmm-agent', u'setup'] exited with 1.
Restarting [u'pmm-agent', u'setup'] in 5 seconds because PMM_AGENT_SIDECAR is enabled ...
TBh now the UI also is not logging with the password !!
1 Like
Please edit your post to remove the password (unless it’s really just some one-time test one).
In --set pmm.secrets.passwords.pmmserverkey="..." you need to provide the api key which you generate through the PMM gui (from what I see you have tried to put a password there since it’s short, api key should be much longer).
So that’s why your agents failed to register, they tried to use the api key, but it was incorrect.
Just make sure you are using at least v1.11.4 of pxc-db chart version since from that version pmmserverkey parameter was added.
As for PMM gui password not sure what happened, but you can try to reset it like this: Changing the Default Admin Password in Docker-Based Deployment of PMM2 - Percona Database Performance Blog
although in this case you will need to exec into pmm pod and I think you will need to update it in the pmm-secret also.
1 Like
Hi @Tomislav_Plavcic So I tried resetting and creating apikey and then using that API keybut still same result
Registering pmm-agent on PMM Server...
Failed to register pmm-agent on PMM Server: invalid username or password
Please check username and password.
helm install my-db --set pmm.enabled=true --set pmm.image.repository="percona/pmm-client" --set pmm.image.tag="2.28.0" --set pmm.serverHost="monitoring-service" --set pmm.secrets.passwords.pmmserverkey="eyJrIjoiZGxPMU1HVDcyYnlrbVBwNWt0Yk9kU3V6aVZ0cnpBN3EiLCJuIjoiZGVvbyIsImlkIjoxfQ==" percona/pxc-db
curl -H "Authorization: Bearer eyJrIjoiZGxPMU1HVDcyYnlrbVBwNWt0Yk9kU3V6aVZ0cnpBN3EiLCJuIjoiZGVvbyIsImlkIjoxfQ==" http://74.220.20.248/graph/api/dashboards/home
{"redirectUri":"d/pmm-home/home-dashboard"}
you can see that the key is right
1 Like
Then I saw that the secret is still different
kubectl get secrets internal-my-db-pxc-db -oyaml
apiVersion: v1
data:
clustercheck: aW5zZWN1cmUtY2x1c3RlcmNoZWNrLXBhc3N3b3Jk
monitor: aW5zZWN1cmUtbW9uaXRvci1wYXNzd29yZA==
operator: aW5zZWN1cmUtb3BlcmF0b3ItcGFzc3dvcmQ=
pmmserver: aW5zZWN1cmUtcG1tc2VydmVyLXBhc3N3b3Jk
proxyadmin: aW5zZWN1cmUtcHJveHlhZG1pbi1wYXNzd29yZA==
replication: aW5zZWN1cmUtcmVwbGljYXRpb24tcGFzc3dvcmQ=
root: aW5zZWN1cmUtcm9vdC1wYXNzd29yZA==
xtrabackup: aW5zZWN1cmUteHRyYWJhY2t1cC1wYXNzd29yZA==
kind: Secret
metadata:
annotations:
grant-for-1.6.0-monitor-user: done
grant-for-1.7.0-xtrabackup-user: done
grant-for-1.10.0-system-privilege: done
creationTimestamp: "2022-07-08T10:19:18Z"
name: internal-my-db-pxc-db
namespace: default
resourceVersion: "16794"
uid: 7a7748ff-39ad-4601-9c9c-2f6d665751ee
type: Opaque
demo ~ echo "aW5zZWN1cmUtcG1tc2VydmVyLXBhc3N3b3Jk"
aW5zZWN1cmUtcG1tc2VydmVyLXBhc3N3b3Jk
demo ~ echo "aW5zZWN1cmUtcG1tc2VydmVyLXBhc3N3b3Jk" | base64 -d
insecure-pmmserver-password%
1 Like
Sorry my bad! I had a typo in the parameters name, the correct one is secrets.passwords.pmmserverkey, but at first I have put pmm.secrets.passwords.pmmserverkey.
So this command has worked for me:
helm install my-db --set pmm.enabled=true --set pmm.image.repository="percona/pmm-client" --set pmm.image.tag="2.28.0" --set pmm.serverHost="monitoring-service" --set secrets.passwords.pmmserverkey="eyJrIjoiSnRuanFDTTU1VW9IN1lYbllzeXJQSjFBMzI4aWRoSmkiLCJuIjoib3BlcmF0b3IiLCJpZCI6MX0=" --version 1.11.5 percona/pxc-db
Please also run helm repo update to make sure you are using the latest versions of the helm charts.
1 Like
Looks like I am close
the error has updated to
hecking local pmm-agent status...
pmm-agent is running.
Registering pmm-agent on PMM Server...
Failed to register pmm-agent on PMM Server: Access denied.
1 Like
Yes, just make sure you are using the proper key in the command, because in my output I have put my test key.
1 Like
perfect! after a few failed attempts I am finally able to see fancy data in the dashboards! thank you for all the help @Tomislav_Plavcic
Loved your support overall!
2 Likes
