Issue Passing Environment Variables to vm_agent in PMM Client

Percona Monitoring and Management (PMM) PMM 2.x
1

Description:

I am encountering an issue with PMM Client where the vm_agent process fails to scrape metrics due to exceeding the default -promscrape.maxScrapeSize limit. I enabled debug mode and captured the following error:

the response size exceeds -promscrape.maxScrapeSize=67108864; either reduce the response size for the target or increase -promscrape.maxScrapeSize.

I have reviewed the Percona documentation on VictoriaMetrics, which mentions the use of environment variables. However, it is unclear whether these can be applied to the PMM Client (pmm-agent) and how to set them for vm_agent.

I tried setting the environment variable both in the pmm-agent binary installation and using the Docker image, but it didn’t work.

Steps to Reproduce:

  1. Set up PMM Client and register the MySQL service using pmm-admin.
  2. Enable debug mode and review the logs when vm_agent is scraping metrics.
  3. Check for the error related to -promscrape.maxScrapeSize when scraping MySQL metrics.
  4. Attempt to set VM_promscrape_maxScrapeSize to a higher value, both in the Docker container and using the binary.

Version:

  • PMM Server: 2.42.0
  • PMM Client: 2.42.0

Logs:

2024-10-18T08:39:06.218+00:00 level=info msg="cannot scrape target... the response size exceeds -promscrape.maxScrapeSize=67108864; either reduce the response size for the target or increase -promscrape.maxScrapeSize" component=agent-process type=vm_agent

Expected Result:

The vm_agent should be able to scrape the metrics successfully without exceeding the maxScrapeSize limit, or the environment variable to increase the block size should be passed and applied to vm_agent.

Actual Result:

vm_agent is unable to scrape metrics due to the size limit being exceeded, and attempts to set promscrape.maxScrapeSize have not worked. It is unclear whether environment variables can be applied to vm_agent running under PMM Client, or if they are only for PMM Server.

Additional Information:

  • The vm_agent process runs with -envflag.prefix=VMAGENT_ and -envflag.enable=true, but setting variables like VMAGENT_promscrape_maxScrapeSize does not seem to have any effect.
  • The documentation does not clarify whether environment variables can be applied on the client side and what prefix (VM_ or VMAGENT_) should be used for vm_agent.
2

Hello @Adrian_Castillo,
Can you provide more details on your setup of pmm-agent? Do you have any custom queries enabled? The out-of-box settings for PMM perfectly allow for scraping the data, so your setup must be different in some way to cause this issue.

3

Hi @matthewb , and thanks for the reply!

I have no custom queries for that service. The setup for the pmm-agent is very simple, I’ll share with you the command for the host registration and the MySQL service config:

pmm-admin config --server-insecure-tls --server-url "https://xxx:xxx@x.x.x.x:xxx"

pmm-admin add mysql --query-source=perfschema --username=xxx --password='xxx' --disable-tablestats-limit=3000

Because of I want all the info about “Table Details” panel, I’ve added the “disable-tablestats-limits” because of the huge number of tables the server haves.

I’m able to collect “Tables Details” and “Users Detaills” metrics (the only two panels that there was no data) removing the service and adding a “disable-collectos” flag:

pmm-admin add mysql --query-source=perfschema --username=xxx --password='xxx' --disable-tablestats-limit=3000 --disable-collectors=info_schema.clientstats

But we want to get all the collectors, that’s why I’m trying to pass the “-promscrape.maxScrapeSize” flag to the vm_agent.

4

Hmm. You should not need to set this, even when enabling all collectors. The defaults handle most situations.

How many? This might be the actual problem. If you have 10,000 tables, that might be too much for the scraper size.

The error messsage you showed above, does that come from PMM server? or the client agent?

5

Hi Adrián,
Please check [PMM-12638] - Percona JIRA

PMM_PROMSCRAPE_MAX_SCRAPE_SIZE should be passed to pmm-server. any env variables on the client side are ignored at the moment.

What PMM server deployment method are you using?

6

Thaks @Agustin_G!

I’m using k8s deployment, and I’ve added the PMM_PROMSCRAPE_MAX_SCRAPE_SIZE inside pmmEnv block.

This approach solved the problem!

1 Like