MySQL Table details error

[PavelVD]

Problem with dashboard Table Details

Templating [service_name]
Error updating options: cannot fetch data for "MinTimestamp=2022-10-18 21:16:45 +0000 UTC, MaxTimestamp=2022-10-19 09:16:45 +0000 UTC, TagFilters=[\n{Key=\"\", Value=\"mysql_info_schema_table_version\", IsNegative: false, IsRegexp: false}\n]": search error after reading 0 data blocks: error when searching for tagFilters=[{__name__="mysql_info_schema_table_version"}] on the time range [2022-10-18 21:16:45 +0000 UTC - 2022-10-19 09:16:45 +0000 UTC]: error when searching tsids: the number of matching timeseries exceeds 10000; either narrow down the search or increase -search.max* command-line flag values at vmselect```

[Michael_Coburn]

Hi @PavelVD welcome back!
Can you share some details about your deployment:

• version of PMM Server and Client
• Version and distribution of MySQL
• Number of instances in this PMM Server

Looks to me like you have a lot of MySQL instances being monitored. If you can fill in some details then we can come back with a tuning recommendation for VictoriaMetrics. Thanks!

[PavelVD]

PMM Server and clients 2.29.0
Mysql different versions
Percona 5.7
MariaDB 10.3
MariaDB 10.6
75 monitoring services

[Michael_Coburn]

Thanks for the feedback. I’m escalating with the PMM Engineers to see if we can provide some guidance here.

[Anton_Bystrov]

@PavelVD

`Error updating options: cannot fetch data for “MinTimestamp=2022-10-18 21:16:45 +0000 UTC, MaxTimestamp=2022-10-19 09:16:45 +0000 UTC, TagFilters=[\n{Key="", Value="mysql_info_schema_table_version", IsNegative: false, IsRegexp: false}\n]”: search error after reading 0 data blocks: error when searching for tagFilters=[{name=“mysql_info_schema_table_version”}] on the time range [2022-10-18 21:16:45 +0000 UTC - 2022-10-19 09:16:45 +0000 UTC]: error when searching tsids: the number of matching timeseries exceeds 10000; either narrow down the search or increase -search.max* command-line flag values at vmselect````

Hello! You have default pmm installation without any changes in VictoriaMetrics? Can you share result “ps -aux | grep victoriametrics” from your instance with pmm-server?

[PavelVD]

@Anton_Bystrov

ps -aux | grep victoriametrics

No any changes in VictoriaMetrics - only default

rsindar+  1847 53.5 42.9 46147580 6977992 ?    Sl   Aug16 56275:55 /usr/sbin/victoriametrics --promscrape.config=/etc/victoriametrics-promscrape.yml --retentionPeriod=30d --storageDataPath=/srv/victoriametrics/data --httpListenAddr=127.0.0.1:9090 --search.disableCache=true --search.maxQueryLen=1MB --search.latencyOffset=5s --search.maxUniqueTimeseries=60000000 --search.maxQueueDuration=30s --search.logSlowQueryDuration=30s --search.maxQueryDuration=60s --promscrape.streamParse=true --prometheusDataPath=/srv/prometheus/data --http.pathPrefix=/prometheus --envflag.enable --envflag.prefix=VM_
root      1966 26.4  0.7 5009648 114632 ?      Sl   Aug16 27797:51 /usr/sbin/pmm-managed --victoriametrics-config=/etc/victoriametrics-promscrape.yml --victoriametrics-url=http://127.0.0.1:9090/prometheus --postgres-name=pmm-managed --postgres-username=pmm-managed --postgres-password=***--supervisord-config-dir=/etc/supervisord.d

[Anton_Bystrov]

Thank you. You have 75 services, right? And how many databases do you have? Did you have any databases growth at 2022-10-18 - 2022-10-19?

[PavelVD]

We constantly make changes in databases, including the number of monitoring objects, it is very difficult to associate with a specific change (

[PavelVD]

And this error on any time frame

Error updating options: cannot fetch data for "MinTimestamp=2022-10-28 03:01:10 +0000 UTC, MaxTimestamp=2022-10-28 15:01:10 +0000 UTC, TagFilters=[\n{Key=\"\", Value=\"mysql_info_schema_table_version\", IsNegative: false, IsRegexp: false}\n]": search error after reading 0 data blocks: error when searching for tagFilters=[{__name__="mysql_info_schema_table_version"}] on the time range [2022-10-28 03:01:10 +0000 UTC - 2022-10-28 15:01:10 +0000 UTC]: error when searching tsids: the number of matching timeseries exceeds 10000; either narrow down the search or increase -search.max* command-line flag values at vmselect

[PavelVD]

About database count - it ~ 1500

[Anton_Bystrov]

@PavelVD

Error updating options: cannot fetch data for "MinTimestamp=2022-10-28 03:01:10 +0000 UTC, MaxTimestamp=2022-10-28 15:01:10 +0000 UTC, TagFilters=[\n{Key=\"\", Value=\"mysql_info_schema_table_version\", IsNegative: false, IsRegexp: false}\n]": search error after reading 0 data blocks: error when searching for tagFilters=[{__name__="mysql_info_schema_table_version"}] on the time range [2022-10-28 03:01:10 +0000 UTC - 2022-10-28 15:01:10 +0000 UTC]: error when searching tsids: the number of matching timeseries exceeds 10000; either narrow down the search or increase -search.max* command-line flag values at vmselect

Looks like you have limitation at “-search.maxSeries”. How you run pmm instance? In docker?

[PavelVD]

Yes, in docker and only default params

[Anton_Bystrov]

You can try to increase -search.maxSeries parameter in your docker container directly (add in /etc/supervisord.d/victoriametrics.ini --search.maxSeries=30000, and supervisorctl reread && supervisorctl stop victoriametrics && supervisorctl remove victoriametrics && supervisorctl add victoriametrics in docker container with pmm) or waiting for new release of PMM. It should be fixed at this PR: PMM-10629: upgrade to VictoriaMetrics 1.82.1 by michal-kralik · Pull Request #1325 · percona/pmm · GitHub

[PavelVD]

Thank you for reply!