Cannot read Prometheus exposition data: cannot read a block of data in 0.000s

Percona Monitoring and Management (PMM)
1

Can someone please help me on the below error?

Dec 13 04:17:15 test pmm-agent: #033[36mINFO#033[0m[2022-12-13T04:17:15.151-08:00] 2022-12-13T12:17:15.150Z#011error#011VictoriaMetrics/lib/promscrape/scrapework.go:355#011error when scraping “http://127.0.0.1:42003/metrics?collect[]=auto_increment.columns&collect[]=binlog_size&collect[]=custom_query.lr&collect[]=engine_tokudb_status&collect[]=global_variables&collect[]=heartbeat&collect[]=info_schema.clientstats&collect[]=info_schema.innodb_tablespaces&collect[]=info_schema.tables&collect[]=info_schema.tablestats&collect[]=info_schema.userstats&collect[]=perf_schema.eventsstatements&collect[]=perf_schema.file_instances&collect[]=perf_schema.indexiowaits&collect[]=perf_schema.tableiowaits” from job “mysqld_exporter_agent_id_d821abf0-d2d3-4934-97e5-3be050b6e248_lr” with labels {APP_TYPE=“QA_ALL_TRACKS”,DB_SERVICE_TYPE=“GALERA_CLUSTER(test/402/403)”,ENV=“C-STACK-QA”,SERVICE_RUNNING=“MariaDB”,agent_id=“/agent_id/d821abf0-d2d3-4934-97e5-3be050b6e248”,agent_type=“mysqld_exporter”,cluster=“C-QA4”,environment=“C-STACK-QA”,instance=“/agent_id/d821abf0-d2d3-4934-97e5-3be050b6e248”,job=“mysqld_exporter_agent_id_d821abf0-d2d3-4934-97e5-3be050b6e248_lr”,machine_id=“/machine_id/4e4c02654583449ea8e09f4b826d25be”,node_id=“/node_id/fcce5539-0b03-4d80-94c9-8cb704afb61c”,node_name=“test”,node_type=“generic”,service_id=“/service_id/0037b4f1-1ede-465a-9eb4-75d180790b20”,service_name=“C-QA4-401”,service_type=“mysql”}: cannot read Prometheus exposition data: cannot read a block of data in 0.000s: the response from “http://127.0.0.1:42003/metrics?collect[]=auto_increment.columns&collect[]=binlog_size&collect[]=custom_query.lr&collect[]=engine_tokudb_status&collect[]=global_variables&collect[]=heartbeat&collect[]=info_schema.clientstats&collect[]=info_schema.innodb_tablespaces&collect[]=info_schema.tables&collect[]=info_schema.tablestats&collect[]=info_schema.userstats&collect[]=perf_schema.eventsstatements&collect[]=perf_schema.file_instances&collect[]=perf_schema.indexiowaits&collect[]=perf_schema.tableiowaits” exceeds -promscrape.maxScrapeSize=16777216; either reduce the response size for the target or increase -promscrape.maxScrapeSize #033[36magentID#033[0m=/agent_id/8e573894-7ea9-4ade-99e1-a8741356f128 #033[36mcomponent#033[0m=agent-process #033[36mtype#033[0m=vm_agent

2

Can someone please help me on this?

3

Can someone please help me on this?

4

Can someone please help me on this?

5

Hello! Can you describe your instances and environment?

6

Hi @Anton_Bystrov

We have MariaDB 10.5 enterprise version with Galera cluster.
Operating System: Centos 7

7

Hello! @Naresh9999 Looks like you have too big response ( promscrape.maxScrapeSize is the
maximum size of scrape response in bytes to process from Prometheus targets. By default it 16777216) May be you have a lot of databases and tables and you turn on some custom queries?

8

Hi @Anton_Bystrov

I have enabled only the slow log and table stats report on that database, If I disable then will that resolve the issue?

9

How many databases and tables do you have?

10

Hi @Anton_Bystrov

I have total 121 databases and tables count around 4000.