PMM2 Custom password for mysqld exporter

[beniamin]

Hello,

I want to be able to scrape metrics exported by mysld_exported from a Prometheus instances that is outside of the PMM.

At the moment the documentation says that the password is autogenerated using the agent-id, but this is a real inconvenient for me because I have a lot of mysql instances and I have to manually define new scrape config for each mysql instance.

I saw in the mysqld_exporter help that it has support for custom authentication using HTTP_AUTH env variable or --web.auth-file=WEB.AUTH-FILE param, but I can’t figure out how to set this param using pmm-admin or pmm-agent. I tried with pmm-admin add mysql --environment HTTP_AUTH=user:pass, but with no luck.

Do you know if it’s possible to do that?

[matthewb]

Hi @beniamin Looking at this commit, https://github.com/percona/pmm-server/blob/master/entrypoint.sh, it looks like SERVER_USER and SERVER_PASSWORD might be what you need.

Looking here, https://github.com/percona/mysqld_exporter/blob/c6544d4742b1bb3986dea7ff17fab9f50fab0d30/mysqld_exporter.go#L89, that seems to be the case.

Try those and let me know what you find. Maybe the documentation is out of date with recent changes.

[beniamin]

Hello,

I tried, but your suggestion is setting a custom user/password on communication between pmm-client(s) and pmm-server.

What I need is to be able to set a custom password for the mysld_exporter metrics webpage, in order to be able to get these metrics from an external(outside of PMM environment) Prometheus instance. By default, when a new mysql service is added using ‘pmm-admin add mysql’ the HTTP_AUTH variable is set using the agent-id and I can’t figure it out how to change this default value.

This is the environment set for mysqld_exporter:

cat environ

DATA_SOURCE_NAME=root:toor@tcp(mysql:3306)/?timeout=1sHTTP_AUTH=pmm:/agent_id/005b7f74-014e-4040-9f97-e4f91f16e0ef

[matthewb]

@beniamin I’m looking directly at the source code for the mysqld_exporter. https://github.com/percona/mysqld_exporter/blob/c6544d4742b1bb3986dea7ff17fab9f50fab0d30/mysqld_exporter.go#L380

Looks like you need to create a yaml file and pass the flag web.auth-fileto the exporter.

Looks like HTTP_AUTH is viable as well: https://github.com/percona/mysqld_exporter/blob/c6544d4742b1bb3986dea7ff17fab9f50fab0d30/mysqld_exporter.go#L377

You will most likely need to set the password for both the exporter, and configure your password within PMM server as noted above as well in order for both PMM to function and your custom thing to function.

[matthewb]

Hi @beniamin Let me know if the above solution worked for you. If so, please mark as answered.

[beniamin]

Unfortunately I didn’t find a way to set web.auth-file/HTTP_AUTH to the exporter trough pmm-admin.

None of the suggested solutions worked. I ended up extracting all agent_ids from all mysql instances and use it as a password for my external prometheus.

[nurlan]

Hi @beniamin,

At the moment we don’t support ability to set custom username/password for exporters when we run them as part of PMM.

Please create a task in our Jira https://jira.percona.com/browse/PMM with explanation why do you need this feature.

Thank you

[matthewb]

Hi @beniamin Your post has caused lots of discussion here. Yes, the exporter itself supports custom un/pw, but only if launched outside of PMM. The ‘pmm-agent’ acts as a supervisor and this is what runs/manages the exporter. Right now, there is no way to pass a custom un/pw config through pmm-agent to pass along to the exporter.

Right now, your options are to do as you’ve done, scraping the agent IDs, or run the exporter by itself, passing the parameter you need and forgo the PMM integration.

[zamolxe]

@beniamin,
You can disable the exporter authentication completely with the below workaround, which will give you the possibility to scrape the percona mysqld_exporter from a 2nd prometheus instance without affecting the PMM2 server integration.

On rhel/centos >= 7 create the file /usr/local/percona/pmm2/exporters/mysqld_exporter_noauth like below

#!/bin/bash
exporter_name=$(basename $0)
export HTTP_AUTH=''
$(dirname $0)/${exporter_name%%_noauth} $@

Update permissions and pmm-config.yml

chown pmm-agent.pmm-agent /usr/local/percona/pmm2/exporters/mysqld_exporter_noauth
chmod 0755 /usr/local/percona/pmm2/exporters/mysqld_exporter_noauth
sed -i -e 's/exporters\/mysqld_exporter$/exporters\/mysqld_exporter_noauth/g' /usr/local/percona/pmm2/config/pmm-agent.yaml
systemctl restart pmm-agent

I hope this helps.
If interested I can add some discovery mechanism using the pmm2 server api when I’m done with it.

/Alex

[beniamin]

Thanks @zamolxe for you suggestion. Indeed this solution may work, but our security team does not allow me to use endpoints without authentication.