ERROR qan-analyzer driver: bad connection

  • Open “Query Analytics” interface
  • choose database host
  • click on SETTINGS button
  • disable (uncheck) “Send real query examples” flag
  • set “Collect from” field to “Performance Schema”
  • click apply button

please notify me if any bugs still exists

photoid=47215

You were right, it work fine with disabled “Send real query examples

Is there some way to disable “Send real query examples” from command line? I just use ansible to add hosts to PMM, so i need to add it without GUI.

currently it is imposible to add node with disabled “Send real query examples” option.

but, I added such functionality as experimental feature (it is not reviewed and not tested) - [url]https://github.com/percona/pmm-client/pull/24[/url]
if you want you can compile it and test


sudo yum install golang git
mkdir /tmp/pmm-admin
cd /tmp/pmm-admin
git clone -b query-examples https://github.com/percona/pmm-client src/github.com/percona/pmm-client
export GOPATH=$(pwd)
go build ./src/github.com/percona/pmm-client/
sudo mv pmm-client /usr/sbin/pmm-admin
sudo pmm-admin remove mysql
sudo pmm-admin add mysql --disable-queryexamples

anyway it is workaround, we are searching root of the issue.

Hi Alexksey,

after all, I created the issue [url][PMM-589] capturing query examples from perf schema freezes the agent - Percona JIRA
feel free to add any additional information to jira ticket.

Ok, thanks.

Adventure does not end. Everything works fine, until slow_query_log disabled. In spite of the fact usage of performance_schema. Same problem, bad connection driver.

sorry, can you clarify one more time.
when slow_query_log=OFF, performance_schema=ON, query_examples=OFF, Do you have any issues?

Right now “slow_query_log=OFF, performance_schema=ON, query_examples=OFF”.
Problem exist.
A lot of WARNING in pmm-mysql-queries-0.log:

2017/02/17 08:51:00.001570 WARNING qan-analyzer-f122d8f8-worker Interval out of sequence: got 85, expected 81
2017/02/17 08:52:00.001470 WARNING qan-analyzer-f122d8f8 Skipping interval ‘86 2017-02-17 06:51:00 UTC to 2017-02-17 06:52:00 UTC (0-0)’ because interval ‘85 2017-02-17 06:50:00 UTC to 2017-02-17 06:51:00 UTC (0-0)’ is still being parsed
2017/02/17 08:53:00.001461 WARNING qan-analyzer-f122d8f8 Skipping interval ‘87 2017-02-17 06:52:00 UTC to 2017-02-17 06:53:00 UTC (0-0)’ because interval ‘85 2017-02-17 06:50:00 UTC to 2017-02-17 06:51:00 UTC (0-0)’ is still being parsed
2017/02/17 08:54:00.001591 WARNING qan-analyzer-f122d8f8 Skipping interval ‘88 2017-02-17 06:53:00 UTC to 2017-02-17 06:54:00 UTC (0-0)’ because interval ‘85 2017-02-17 06:50:00 UTC to 2017-02-17 06:51:00 UTC (0-0)’ is still being parsed
2017/02/17 08:55:00.001422 WARNING qan-analyzer-f122d8f8 Skipping interval ‘89 2017-02-17 06:54:00 UTC to 2017-02-17 06:55:00 UTC (0-0)’ because interval ‘85 2017-02-17 06:50:00 UTC to 2017-02-17 06:51:00 UTC (0-0)’ is still being parsed
2017/02/17 08:56:00.001660 WARNING qan-analyzer-f122d8f8-worker Interval out of sequence: got 90, expected 86
2017/02/17 08:57:00.001457 WARNING qan-analyzer-f122d8f8 Skipping interval ‘91 2017-02-17 06:56:00 UTC to 2017-02-17 06:57:00 UTC (0-0)’ because interval ‘90 2017-02-17 06:55:00 UTC to 2017-02-17 06:56:00 UTC (0-0)’ is still being parsed

After manipulation on settings page - switch to slowlog, switch to performance_schema, set and unset Send real query examples QAN start to work.

these warnings just warnings and shouldn’t affect qan-api functionality.

If i need to reinstall pmm-client, how do i have to clear all settings, caches, tampfiles etc after removing and before installing? Is there some place where all of this stored?

before reinstall please type pmm-admin uninstall on client side.

Ok, i removed, installed and configured client again, will see. Can i write here problems with grafana, or create other topic?

I think new topic is better

[url]https://www.percona.com/forums/questions-discussions/percona-monitoring-and-management/47254-pmm-gravana-does-not-show-all-data[/url]

Well, for now QAN usable only with slow_query_log, not with performance_schema. Tested on 1.1.1 version.

what issues do you have with “performance_schema=ON, query_examples=OFF” ?

2017/02/21 10:18:23.795575 main.go:207: API is ready
2017/02/21 11:01:31.792736 WARNING data-sender Waiting for API to ack qan_1487667661266859145: read tcp 192.168.15.9:45680->192.168.15.19:82: i/o timeout
2017/02/21 11:05:00.003461 WARNING qan-analyzer-04bce813-worker Interval out of sequence: got 1, expected 47
2017/02/21 11:06:00.004050 WARNING qan-analyzer-04bce813 Skipping interval ‘2 2017-02-21 09:05:00 UTC to 2017-02-21 09:06:00 UTC (0-0)’ because interval ‘1 0001-01-01 00:00:00 UTC to 2017-02-21 09:05:00 UTC (0-0)’ is still being parsed
2017/02/21 11:07:00.001403 WARNING qan-analyzer-04bce813 Skipping interval ‘3 2017-02-21 09:06:00 UTC to 2017-02-21 09:07:00 UTC (0-0)’ because interval ‘1 0001-01-01 00:00:00 UTC to 2017-02-21 09:05:00 UTC (0-0)’ is still being parsed
2017/02/21 11:08:00.018888 WARNING qan-analyzer-04bce813 Skipping interval ‘4 2017-02-21 09:07:00 UTC to 2017-02-21 09:08:00 UTC (0-0)’ because interval ‘1 0001-01-01 00:00:00 UTC to 2017-02-21 09:05:00 UTC (0-0)’ is still being parsed
2017/02/21 11:09:00.001833 WARNING qan-analyzer-04bce813 Skipping interval ‘5 2017-02-21 09:08:00 UTC to 2017-02-21 09:09:00 UTC (0-0)’ because interval ‘1 0001-01-01 00:00:00 UTC to 2017-02-21 09:05:00 UTC (0-0)’ is still being parsed
2017/02/21 11:09:23.213428 analyzer.go:426: qan-analyzer-04bce813-worker crashed: ‘1 0001-01-01 00:00:00 UTC to 2017-02-21 09:05:00 UTC (0-0)’: runtime error: invalid memory address or nil pointer dereference
goroutine 498 [running]:
runtime/debug.Stack(0x4868ec, 0xc42000e0f0, 0x2)
/usr/local/go1.7.4/src/runtime/debug/stack.go:24 +0x79
runtime/debug.PrintStack()
/usr/local/go1.7.4/src/runtime/debug/stack.go:16 +0x22
github.com/percona/qan-agent/qan.(*RealAnalyzer).runWorker.func1(0xc42016cc60, 0xc420a56ea0)
/mnt/workspace/pmm-client-tarball/pmm-client-1.1.1/src/github.com/percona/qan-agent/qan/analyzer.go:427 +0x1f6
panic(0x7176c0, 0xc42000c070)
/usr/local/go1.7.4/src/runtime/panic.go:458 +0x243
database/sql.(*DB).conn(0x0, 0x7f0aae2bc001, 0x0, 0x1100000000, 0xc420045970)
/usr/local/go1.7.4/src/database/sql/sql.go:781 +0x3a
database/sql.(*DB).query(0x0, 0xc420d36c60, 0xa2, 0x0, 0x0, 0x0, 0x1, 0x10, 0x6fe4c0, 0xc420d36c60)
/usr/local/go1.7.4/src/database/sql/sql.go:1074 +0x3b
database/sql.(*DB).Query(0x0, 0xc420d36c60, 0xa2, 0x0, 0x0, 0x0, 0xa2, 0xc4200459c8, 0x4142f8)
/usr/local/go1.7.4/src/database/sql/sql.go:1062 +0x90
database/sql.(*DB).QueryRow(0x0, 0xc420d36c60, 0xa2, 0x0, 0x0, 0x0, 0xa2)
/usr/local/go1.7.4/src/database/sql/sql.go:1143 +0x70
github.com/percona/qan-agent/qan/perfschema.GetDigestText(0x8dca20, 0xc42021b860, 0xc4216d0080, 0x20, 0xc4216d0090, 0x10, 0xc4215ac950, 0x10)
/mnt/workspace/pmm-client-tarball/pmm-client-1.1.1/src/github.com/percona/qan-agent/qan/perfschema/worker.go:188 +0x164
github.com/percona/qan-agent/qan/perfschema.(*RealWorkerFactory).Make.func2(0xc4216d0080, 0x20, 0xc4219cdc80, 0x10, 0x9140e0, 0x0)
/mnt/workspace/pmm-client-tarball/pmm-client-1.1.1/src/github.com/percona/qan-agent/qan/perfschema/worker.go:107 +0x47
github.com/percona/qan-agent/qan/perfschema.(*Worker).getSnapshot(0xc4200e8f00, 0xc4208e31a0, 0x0, 0x0, 0x0)
/mnt/workspace/pmm-client-tarball/pmm-client-1.1.1/src/github.com/percona/qan-agent/qan/perfschema/worker.go:430 +0x92f
github.com/percona/qan-agent/qan/perfschema.(*Worker).Run(0xc4200e8f00, 0x0, 0x0, 0x0)
/mnt/workspace/pmm-client-tarball/pmm-client-1.1.1/src/github.com/percona/qan-agent/qan/perfschema/worker.go:276 +0x38a
github.com/percona/qan-agent/qan.(*RealAnalyzer).runWorker(0xc42016cc60, 0xc420a56ea0)
/mnt/workspace/pmm-client-tarball/pmm-client-1.1.1/src/github.com/percona/qan-agent/qan/analyzer.go:451 +0x305
created by github.com/percona/qan-agent/qan.(*RealAnalyzer).run
/mnt/workspace/pmm-client-tarball/pmm-client-1.1.1/src/github.com/percona/qan-agent/qan/analyzer.go:365 +0x832
2017/02/21 11:09:23.213692 ERROR qan-analyzer-04bce813 qan-analyzer-04bce813-worker crashed: ‘1 0001-01-01 00:00:00 UTC to 2017-02-21 09:05:00 UTC (0-0)’: runtime error: invalid memory address or nil pointer dereference
2017/02/21 11:10:00.001421 WARNING qan-analyzer-04bce813-worker Interval out of sequence: got 1, expected 2
2017/02/21 11:11:00.001888 WARNING qan-analyzer-04bce813 Skipping interval ‘2 2017-02-21 09:10:00 UTC to 2017-02-21 09:11:00 UTC (0-0)’ because interval ‘1 0001-01-01 00:00:00 UTC to 2017-02-21 09:10:00 UTC (0-0)’ is still being parsed
2017/02/21 11:12:00.001400 WARNING qan-analyzer-04bce813 Skipping interval ‘3 2017-02-21 09:11:00 UTC to 2017-02-21 09:12:00 UTC (0-0)’ because interval ‘1 0001-01-01 00:00:00 UTC to 2017-02-21 09:10:00 UTC (0-0)’ is still being parsed

Other error, but same result - it does not work with performance_schema for a long time. Less then 1 hour. But it work fine with slow query log.
Buy the way, when you advice me in first time switching to performance schema and disabling query sampling, i forgot to disable slow query log in database. And it works. But after disabling slow query log, without changing configuration in QAN, it stop working in 1 hour or so.

thank you, I added this stack trace to developers ticket!