2017/02/21 10:18:23.795575 main.go:207: API is ready
2017/02/21 11:01:31.792736 WARNING data-sender Waiting for API to ack qan_1487667661266859145: read tcp 192.168.15.9:45680->192.168.15.19:82: i/o timeout
2017/02/21 11:05:00.003461 WARNING qan-analyzer-04bce813-worker Interval out of sequence: got 1, expected 47
2017/02/21 11:06:00.004050 WARNING qan-analyzer-04bce813 Skipping interval ‘2 2017-02-21 09:05:00 UTC to 2017-02-21 09:06:00 UTC (0-0)’ because interval ‘1 0001-01-01 00:00:00 UTC to 2017-02-21 09:05:00 UTC (0-0)’ is still being parsed
2017/02/21 11:07:00.001403 WARNING qan-analyzer-04bce813 Skipping interval ‘3 2017-02-21 09:06:00 UTC to 2017-02-21 09:07:00 UTC (0-0)’ because interval ‘1 0001-01-01 00:00:00 UTC to 2017-02-21 09:05:00 UTC (0-0)’ is still being parsed
2017/02/21 11:08:00.018888 WARNING qan-analyzer-04bce813 Skipping interval ‘4 2017-02-21 09:07:00 UTC to 2017-02-21 09:08:00 UTC (0-0)’ because interval ‘1 0001-01-01 00:00:00 UTC to 2017-02-21 09:05:00 UTC (0-0)’ is still being parsed
2017/02/21 11:09:00.001833 WARNING qan-analyzer-04bce813 Skipping interval ‘5 2017-02-21 09:08:00 UTC to 2017-02-21 09:09:00 UTC (0-0)’ because interval ‘1 0001-01-01 00:00:00 UTC to 2017-02-21 09:05:00 UTC (0-0)’ is still being parsed
2017/02/21 11:09:23.213428 analyzer.go:426: qan-analyzer-04bce813-worker crashed: ‘1 0001-01-01 00:00:00 UTC to 2017-02-21 09:05:00 UTC (0-0)’: runtime error: invalid memory address or nil pointer dereference
goroutine 498 [running]:
runtime/debug.Stack(0x4868ec, 0xc42000e0f0, 0x2)
/usr/local/go1.7.4/src/runtime/debug/stack.go:24 +0x79
runtime/debug.PrintStack()
/usr/local/go1.7.4/src/runtime/debug/stack.go:16 +0x22
github.com/percona/qan-agent/qan.(*RealAnalyzer).runWorker.func1(0xc42016cc60, 0xc420a56ea0)
/mnt/workspace/pmm-client-tarball/pmm-client-1.1.1/src/github.com/percona/qan-agent/qan/analyzer.go:427 +0x1f6
panic(0x7176c0, 0xc42000c070)
/usr/local/go1.7.4/src/runtime/panic.go:458 +0x243
database/sql.(*DB).conn(0x0, 0x7f0aae2bc001, 0x0, 0x1100000000, 0xc420045970)
/usr/local/go1.7.4/src/database/sql/sql.go:781 +0x3a
database/sql.(*DB).query(0x0, 0xc420d36c60, 0xa2, 0x0, 0x0, 0x0, 0x1, 0x10, 0x6fe4c0, 0xc420d36c60)
/usr/local/go1.7.4/src/database/sql/sql.go:1074 +0x3b
database/sql.(*DB).Query(0x0, 0xc420d36c60, 0xa2, 0x0, 0x0, 0x0, 0xa2, 0xc4200459c8, 0x4142f8)
/usr/local/go1.7.4/src/database/sql/sql.go:1062 +0x90
database/sql.(*DB).QueryRow(0x0, 0xc420d36c60, 0xa2, 0x0, 0x0, 0x0, 0xa2)
/usr/local/go1.7.4/src/database/sql/sql.go:1143 +0x70
github.com/percona/qan-agent/qan/perfschema.GetDigestText(0x8dca20, 0xc42021b860, 0xc4216d0080, 0x20, 0xc4216d0090, 0x10, 0xc4215ac950, 0x10)
/mnt/workspace/pmm-client-tarball/pmm-client-1.1.1/src/github.com/percona/qan-agent/qan/perfschema/worker.go:188 +0x164
github.com/percona/qan-agent/qan/perfschema.(*RealWorkerFactory).Make.func2(0xc4216d0080, 0x20, 0xc4219cdc80, 0x10, 0x9140e0, 0x0)
/mnt/workspace/pmm-client-tarball/pmm-client-1.1.1/src/github.com/percona/qan-agent/qan/perfschema/worker.go:107 +0x47
github.com/percona/qan-agent/qan/perfschema.(*Worker).getSnapshot(0xc4200e8f00, 0xc4208e31a0, 0x0, 0x0, 0x0)
/mnt/workspace/pmm-client-tarball/pmm-client-1.1.1/src/github.com/percona/qan-agent/qan/perfschema/worker.go:430 +0x92f
github.com/percona/qan-agent/qan/perfschema.(*Worker).Run(0xc4200e8f00, 0x0, 0x0, 0x0)
/mnt/workspace/pmm-client-tarball/pmm-client-1.1.1/src/github.com/percona/qan-agent/qan/perfschema/worker.go:276 +0x38a
github.com/percona/qan-agent/qan.(*RealAnalyzer).runWorker(0xc42016cc60, 0xc420a56ea0)
/mnt/workspace/pmm-client-tarball/pmm-client-1.1.1/src/github.com/percona/qan-agent/qan/analyzer.go:451 +0x305
created by github.com/percona/qan-agent/qan.(*RealAnalyzer).run
/mnt/workspace/pmm-client-tarball/pmm-client-1.1.1/src/github.com/percona/qan-agent/qan/analyzer.go:365 +0x832
2017/02/21 11:09:23.213692 ERROR qan-analyzer-04bce813 qan-analyzer-04bce813-worker crashed: ‘1 0001-01-01 00:00:00 UTC to 2017-02-21 09:05:00 UTC (0-0)’: runtime error: invalid memory address or nil pointer dereference
2017/02/21 11:10:00.001421 WARNING qan-analyzer-04bce813-worker Interval out of sequence: got 1, expected 2
2017/02/21 11:11:00.001888 WARNING qan-analyzer-04bce813 Skipping interval ‘2 2017-02-21 09:10:00 UTC to 2017-02-21 09:11:00 UTC (0-0)’ because interval ‘1 0001-01-01 00:00:00 UTC to 2017-02-21 09:10:00 UTC (0-0)’ is still being parsed
2017/02/21 11:12:00.001400 WARNING qan-analyzer-04bce813 Skipping interval ‘3 2017-02-21 09:11:00 UTC to 2017-02-21 09:12:00 UTC (0-0)’ because interval ‘1 0001-01-01 00:00:00 UTC to 2017-02-21 09:10:00 UTC (0-0)’ is still being parsed
Other error, but same result - it does not work with performance_schema for a long time. Less then 1 hour. But it work fine with slow query log.
Buy the way, when you advice me in first time switching to performance schema and disabling query sampling, i forgot to disable slow query log in database. And it works. But after disabling slow query log, without changing configuration in QAN, it stop working in 1 hour or so.