hello weber,
i wanna know how to how to force enable tablestats=OFF to ON .
currently i have 1000* count tables.
Hello,
[root@centos7 ~]# pmm-admin add mysql:metrics --help
…
–disable-tablestats disable table statistics
–disable-tablestats-limit uint16 number of tables after which table stats are disabled automatically (default 1000)
…
So running the command like this will enforce to enable tablestats:
pmm-admin add mysql:metrics --disable-tablestats-limit 99999 …
hello weber,
definitely if there are too many tables 1000+ , increasing cpu load and runtime error: invalid memory, open queries.
pmm-mysql-queries-0.log
2016/12/23 08:14:15.018841 ERROR qan-analyzer-c4996376 qan-analyzer-c4996376-worker crashed: ‘623 2016-12-22 23:12:00 UTC to 2016-12-22 23:13:00 UTC (0-0)’: runtime error: invalid memory address or nil pointer dereference
User ‘pmm’ has exceeded the ‘max_user_connections’ resource (current value: 10)
additionaly
how can i rearrange Skipping interval ? grafana shows graph lines breaks continuously.
2016/12/23 16:13:00.000936 WARNING qan-analyzer-c8732195 Skipping interval ‘30 2016-12-23 07:12:00 UTC to 2016-12-23 07:13:00 UTC (0-0)’ because interval ‘29 2016-12-23 07:11:00 UTC to 2016-12-23 07:12:00 UTC (0-0)’ is still being parsed
2016/12/23 16:13:00.000994 WARNING qan-analyzer-f84a508e Skipping interval ‘30 2016-12-23 07:12:00 UTC to 2016-12-23 07:13:00 UTC (0-0)’ because interval ‘29 2016-12-23 07:11:00 UTC to 2016-12-23 07:12:00 UTC (0-0)’ is still being parsed
2016/12/23 16:14:00.000839 WARNING qan-analyzer-c8732195-worker Interval out of sequence: got 31, expected 30
2016/12/23 16:14:00.000952 WARNING qan-analyzer-f84a508e-worker Interval out of sequence: got 31, expected 30
From most relevant to lower:
- tablestats off
- increase memory limit for Prometheus (see [url]Percona Monitoring and Management), by default METRICS_MEMORY is set to 256 Mb. You need to increase it with 3+ clients. Say for 10 clients, it should be at least 2-3 Gb.
- finally, if there is a network latency, you may want to lower metrics resolution from 1s to 5s (METRICS_RESOLUTION variable).
despite resources increase, skipping interval warning still occur in components such as memory and processors.
prometheus/flags
storage.local.memory-chunks 6291456
2016/12/28 13:59:00.001678 WARNING qan-analyzer-a34f8701 Skipping interval ‘62 2016-12-28 04:58:00 UTC to 2016-12-28 04:59:00 UTC (0-0)’ because interval ‘61 2016-12-28 04:57:00 UTC to 2016-12-28 04:58:00 UTC (0-0)’ is still being parsed
2016/12/28 13:59:00.001871 WARNING qan-analyzer-35d40036-worker Interval out of sequence: got 49, expected 48
Hi scar2yjs,
Have you tried to reduce metrics resolution from 1s to 5s (METRICS_RESOLUTION variable) ?
“Interval out of sequence: got 49, expected 48” it is just warning, it cannot produce any lines breaks in graphs or any bad consequences.
hello,
unfortunately skipping interval warning still occur again.
16g mem
4 cpu
metrcis 10485760
resolution:
global: scrape_interval: 5s scrape_timeout: 5s evaluation_interval: 5s
0 UTC to 2016-12-29 05:52:00 UTC (0-0)’ is still being parsed
2016/12/29 14:54:00.000958 WARNING qan-analyzer-9723ca8d-worker Interval out of sequence: got 29, expected 28
2016/12/29 14:55:00.000886 WARNING qan-analyzer-9723ca8d Skipping interval ‘30 2016-12-29 05:54:00 UTC to 2016-12-29 05:55:00 UTC (0-0)’ because interval ‘29 2016-12-29 05:53:00 UTC to 2016-12-29 05:54:00 UTC (0-0)’ is still being parsed
-
I have deeply inspected qan-agent code, I am sure “Skipping interval” errors cannot produce any gaps on grafana graphs.
Looks like you database server is overloaded (not PMM server).
Anyway, Why you need to fix these warnings? What bad consequences do you have? -
To change collect interval in qan-agent from 1 minute to 5 minutes it is needed to login to Percona Query Analytics Web Interface, press “SETTINGS” button and change “Collect interval” value".
