got error when execute pmm-admin purge

[Fan]

via docker
1.13.0

[root@node002217 fanboshi]# pmm-admin purge mysql:metrics 192.168.2.217_3307
panic: interface conversion: interface {} is nil, not map[string]interface {}

goroutine 1 [running]:
github.com/percona/pmm-client/pmm.(*Admin).PurgeMetrics(0xbe4e60, 0x7ffc1c4d28d6, 0xd, 0xc4202204a8, 0xc4201cbb68, 0x7ff262)
/tmp/go/src/github.com/percona/pmm-client/pmm/main.go:427 +0x5a4
main.glob…func39(0xbde460, 0xc42019d4c0, 0x2, 0x2)
/tmp/go/src/github.com/percona/pmm-client/pmm-admin.go:1179 +0xd1
github.com/percona/pmm-client/vendor/github.com/spf13/cobra.(*Command).execute(0xbde460, 0xc42019d440, 0x2, 0x2, 0xbde460, 0xc42019d440)
/tmp/go/src/github.com/percona/pmm-client/vendor/github.com/spf13/cobra/command.go:654 +0x2a2
github.com/percona/pmm-client/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xbdeb20, 0xc4201cbdd0, 0x49b302, 0x0)
/tmp/go/src/github.com/percona/pmm-client/vendor/github.com/spf13/cobra/command.go:729 +0x2d4
github.com/percona/pmm-client/vendor/github.com/spf13/cobra.(*Command).Execute(0xbdeb20, 0xc0263c, 0x92bf0f)
/tmp/go/src/github.com/percona/pmm-client/vendor/github.com/spf13/cobra/command.go:688 +0x2b
main.main()
/tmp/go/src/github.com/percona/pmm-client/pmm-admin.go:1398 +0x18dc

[Michael_Coburn]

Hi Fan

Can you run this with verbose option:

pmm-admin --verbose ...

Also, can you attach the logs:

/var/log/pmm-mysql*.log

[Fan]

@Michael Coburn;51988

Hi Fan

Can you run this with verbose option:

pmm-admin --verbose ...

Also, can you attach the logs:

/var/log/pmm-mysql*.log

this time i purge linux:metrics (don’t support markdown, so the format is terrible)
[root@node002142 fanboshi]# pmm-admin add linux:metrics
OK, now monitoring this system.
[root@node002142 fanboshi]# pmm-admin uninstall
OK, 1 services were removed.
[root@node002142 fanboshi]# pmm-admin purge linux:metrics
panic: interface conversion: interface {} is nil, not map[string]interface {}

goroutine 1 [running]:
github.com/percona/pmm-client/pmm.(*Admin).PurgeMetrics(0xbe4e60, 0x7ffe280718e9, 0xd, 0xc420220718, 0xc4201cbb68, 0x7ff262)
/tmp/go/src/github.com/percona/pmm-client/pmm/main.go:427 +0x5a4
main.glob…func39(0xbde460, 0xc4200e56b0, 0x1, 0x1)
/tmp/go/src/github.com/percona/pmm-client/pmm-admin.go:1179 +0xd1
github.com/percona/pmm-client/vendor/github.com/spf13/cobra.(*Command).execute(0xbde460, 0xc4200e5660, 0x1, 0x1, 0xbde460, 0xc4200e5660)
/tmp/go/src/github.com/percona/pmm-client/vendor/github.com/spf13/cobra/command.go:654 +0x2a2
github.com/percona/pmm-client/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xbdeb20, 0xc4201cbdd0, 0x49b302, 0x0)
/tmp/go/src/github.com/percona/pmm-client/vendor/github.com/spf13/cobra/command.go:729 +0x2d4
github.com/percona/pmm-client/vendor/github.com/spf13/cobra.(*Command).Execute(0xbdeb20, 0xc0263c, 0x92bf0f)
/tmp/go/src/github.com/percona/pmm-client/vendor/github.com/spf13/cobra/command.go:688 +0x2b
main.main()
/tmp/go/src/github.com/percona/pmm-client/pmm-admin.go:1398 +0x18dc
[root@node002142 fanboshi]# pmm-admin --verbose purge linux:metrics
2018/08/04 12:51:18 request:

GET /qan-api/ping HTTP/1.1
Host: 10.1.2.143:8080
User-Agent: Go-http-client/1.1
Authorization: Basic bWZ3X2RiYTpacWRSSGV2bmE4
Accept-Encoding: gzip

2018/08/04 12:51:18 response:
< HTTP/1.1 200 OK
< Access-Control-Allow-Headers: Content-Type,Authorization
< Access-Control-Allow-Methods: GET,PUT,POST,DELETE
< Access-Control-Allow-Origin: *
< Cache-Control: no-store
< Connection: keep-alive
< Content-Type: text/plain; charset=utf-8
< Date: Sat, 04 Aug 2018 04:51:18 GMT
< Pragma: no-cache
< Server: nginx
< Strict-Transport-Security: max-age=63072000; includeSubDomains
< X-Content-Type-Options: nosniff
< X-Frame-Options: DENY
< X-Percona-Qan-Api-Version: 1.13.0-1807241501.89a7875
< X-Xss-Protection: 1; mode=block
< Content-Length: 0
<
<
2018/08/04 12:51:18 request:

GET /v1/status/leader HTTP/1.1
Host: 10.1.2.143:8080
User-Agent: Go-http-client/1.1
Authorization: Basic bWZ3X2RiYTpacWRSSGV2bmE4
Accept-Encoding: gzip

2018/08/04 12:51:18 response:
< HTTP/1.1 200 OK
< Content-Length: 16
< Connection: keep-alive
< Content-Type: application/json
< Date: Sat, 04 Aug 2018 04:51:18 GMT
< Server: nginx
< X-Remote-Ip: 10.1.2.142
< X-Server-Time: Saturday, 04-Aug-2018 04:51:18 GMT
<
< “127.0.0.1:8300”
2018/08/04 12:51:18 request:

GET /qan-api/ping HTTP/1.1
Host: 10.1.2.143:8080
User-Agent: Go-http-client/1.1
Accept-Encoding: gzip

2018/08/04 12:51:18 response:
< HTTP/1.1 401 Unauthorized
< Content-Length: 188
< Connection: keep-alive
< Content-Type: text/html
< Date: Sat, 04 Aug 2018 04:51:18 GMT
< Server: nginx
< Www-Authenticate: Basic realm=“on”
<
<
< 401 Authorization Required
<
<

401 Authorization Required

<

---

nginx
<
<
<
2018/08/04 12:51:18 request:

GET /v1/catalog/node/192.168.2.142 HTTP/1.1
Host: 10.1.2.143:8080
User-Agent: Go-http-client/1.1
Authorization: Basic bWZ3X2RiYTpacWRSSGV2bmE4
Accept-Encoding: gzip

2018/08/04 12:51:18 response:
< HTTP/1.1 200 OK
< Content-Length: 170
< Connection: keep-alive
< Content-Type: application/json
< Date: Sat, 04 Aug 2018 04:51:18 GMT
< Server: nginx
< X-Consul-Index: 4680
< X-Consul-Knownleader: true
< X-Consul-Lastcontact: 0
< X-Remote-Ip: 10.1.2.142
< X-Server-Time: Saturday, 04-Aug-2018 04:51:18 GMT
<
< {“Node”:{“ID”:"",“Node”:“192.168.2.142”,“Address”:“10.1.2.142”,“Datacenter”:“dc1”,“TaggedAddresses”:null,“Meta”:null,“CreateIndex”:4678,“ModifyIndex”:4678},“Services”:{}}
2018/08/04 12:51:18 request:

DELETE /prometheus/api/v1/series?match[]={job=“linux”,instance=“192.168.2.142”} HTTP/1.1
Host: 10.1.2.143:8080
User-Agent: Go-http-client/1.1
Authorization: Basic bWZ3X2RiYTpacWRSSGV2bmE4
Accept-Encoding: gzip

2018/08/04 12:51:18 response:
< HTTP/1.1 500 Internal Server Error
< Access-Control-Allow-Headers: Accept, Authorization, Content-Type, Origin
< Access-Control-Allow-Methods: GET, OPTIONS
< Access-Control-Allow-Origin: *
< Access-Control-Expose-Headers: Date
< Connection: keep-alive
< Content-Type: application/json
< Date: Sat, 04 Aug 2018 04:51:18 GMT
< Server: nginx
<
< {“status”:“error”,“errorType”:“internal”,“error”:“not implemented”}
panic: interface conversion: interface {} is nil, not map[string]interface {}

goroutine 1 [running]:
github.com/percona/pmm-client/pmm.(*Admin).PurgeMetrics(0xbe4e60, 0x7ffc91ae58e9, 0xd, 0xc420220648, 0xc4201cbb68, 0x7ff262)
/tmp/go/src/github.com/percona/pmm-client/pmm/main.go:427 +0x5a4
main.glob…func39(0xbde460, 0xc42019d480, 0x1, 0x2)
/tmp/go/src/github.com/percona/pmm-client/pmm-admin.go:1179 +0xd1
github.com/percona/pmm-client/vendor/github.com/spf13/cobra.(*Command).execute(0xbde460, 0xc42019d440, 0x2, 0x2, 0xbde460, 0xc42019d440)
/tmp/go/src/github.com/percona/pmm-client/vendor/github.com/spf13/cobra/command.go:654 +0x2a2
github.com/percona/pmm-client/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xbdeb20, 0xc4201cbdd0, 0x49b302, 0x0)
/tmp/go/src/github.com/percona/pmm-client/vendor/github.com/spf13/cobra/command.go:729 +0x2d4
github.com/percona/pmm-client/vendor/github.com/spf13/cobra.(*Command).Execute(0xbdeb20, 0xc0263c, 0x92bf0f)
/tmp/go/src/github.com/percona/pmm-client/vendor/github.com/spf13/cobra/command.go:688 +0x2b
main.main()
/tmp/go/src/github.com/percona/pmm-client/pmm-admin.go:1398 +0x18dc

log
[root@node002142 log]# less /var/log/pmm-linux-metrics-42000.log
time=“2018-08-04T12:53:21+08:00” level=info msg=“Starting node_exporter (version=, branch=, revision=)” source=“node_exporter.go:142”
time=“2018-08-04T12:53:21+08:00” level=info msg=“Build context (go=go1.10.1, user=, date=)” source=“node_exporter.go:143”
time=“2018-08-04T12:53:21+08:00” level=info msg=“Enabled collectors:” source=“node_exporter.go:162”
time=“2018-08-04T12:53:21+08:00” level=info msg=" - diskstats" source=“node_exporter.go:164”
time=“2018-08-04T12:53:21+08:00” level=info msg=" - filefd" source=“node_exporter.go:164”
time=“2018-08-04T12:53:21+08:00” level=info msg=" - filesystem" source=“node_exporter.go:164”
time=“2018-08-04T12:53:21+08:00” level=info msg=" - meminfo" source=“node_exporter.go:164”
time=“2018-08-04T12:53:21+08:00” level=info msg=" - netdev" source=“node_exporter.go:164”
time=“2018-08-04T12:53:21+08:00” level=info msg=" - uname" source=“node_exporter.go:164”
time=“2018-08-04T12:53:21+08:00” level=info msg=" - meminfo_numa" source=“node_exporter.go:164”
time=“2018-08-04T12:53:21+08:00” level=info msg=" - loadavg" source=“node_exporter.go:164”
time=“2018-08-04T12:53:21+08:00” level=info msg=" - netstat" source=“node_exporter.go:164”
time=“2018-08-04T12:53:21+08:00” level=info msg=" - stat" source=“node_exporter.go:164”
time=“2018-08-04T12:53:21+08:00” level=info msg=" - time" source=“node_exporter.go:164”
time=“2018-08-04T12:53:21+08:00” level=info msg=" - vmstat" source=“node_exporter.go:164”
time=“2018-08-04T12:53:21+08:00” level=info msg=“HTTP Basic authentication is enabled.” source=“basic_auth.go:105”
time=“2018-08-04T12:53:21+08:00” level=info msg=“Starting HTTPS server for https://10.1.2.142:42000/metrics …” source=“server.go:106”

[Fan]

@Michael Coburn;51988

Hi Fan

Can you run this with verbose option:

pmm-admin --verbose ...

Also, can you attach the logs:

/var/log/pmm-mysql*.log

so? Is it a bug or something?

[Fan]

Has this bug been fixed in 1.14? @Michael Coburn

[andycutright]

https://www.percona.com/forums/member/22577-michael-coburn

I see the same issue. I’m running the OVA and MySQL community server on the the client. All the pmm services are running properly on both boxes, as far as I can tell. I removed the linux:metric service, and then purged. I don’t see a log file in /var/log for the linux:metrics service.

sudo pmm-admin --verbose purge linux:metrics

Some data has been elided for security

sudo pmm-admin --verbose purge linux:metrics
2018/09/13 19:53:51 request:
2018/09/13 19:53:51 request:

GET /qan-api/ping HTTP/1.1
Host: 10.0.0.29
User-Agent: Go-http-client/1.1
Authorization: Basic ELIDED
Accept-Encoding: gzip

2018/09/13 19:53:51 response:
< HTTP/1.1 200 OK
< Access-Control-Allow-Headers: Content-Type,Authorization
< Access-Control-Allow-Methods: GET,PUT,POST,DELETE
< Access-Control-Allow-Origin: *
< Cache-Control: no-store
< Connection: keep-alive
< Content-Type: text/plain; charset=utf-8
< Date: Thu, 13 Sep 2018 11:07:11 GMT
< Pragma: no-cache
< Server: nginx
< Strict-Transport-Security: max-age=63072000; includeSubDomains
< X-Content-Type-Options: nosniff
< X-Frame-Options: DENY
< X-Percona-Qan-Api-Version: 1.14.1-1809060744.89a7875
< X-Xss-Protection: 1; mode=block
< Content-Length: 0
<
<
2018/09/13 19:53:51 request:

GET /v1/status/leader HTTP/1.1
Host: 10.0.0.29
User-Agent: Go-http-client/1.1
Authorization: Basic
Accept-Encoding: gzip

2018/09/13 19:53:51 response:
< HTTP/1.1 200 OK
< Content-Length: 16
< Connection: keep-alive
< Content-Type: application/json
< Date: Thu, 13 Sep 2018 11:07:11 GMT
< Server: nginx
< X-Remote-Ip: 10.0.0.200
< X-Server-Time: Thursday, 13-Sep-2018 11:07:11 GMT
<
< “127.0.0.1:8300”
2018/09/13 19:53:51 request:

GET /qan-api/ping HTTP/1.1
Host: 10.0.0.29
User-Agent: Go-http-client/1.1
Accept-Encoding: gzip

2018/09/13 19:53:51 response:
< HTTP/1.1 401 Unauthorized
< Content-Length: 188
< Connection: keep-alive
< Content-Type: text/html
< Date: Thu, 13 Sep 2018 11:07:11 GMT
< Server: nginx
< Www-Authenticate: Basic realm=“on”
<
<
< 401 Authorization Required
<
<

401 Authorization Required

<

---

nginx
<
<
<
2018/09/13 19:53:51 request:

GET /v1/catalog/node/master-stewie HTTP/1.1
Host: 10.0.0.29
User-Agent: Go-http-client/1.1
Authorization: Basic
Accept-Encoding: gzip

2018/09/13 19:53:51 response:
< HTTP/1.1 200 OK
< Content-Length: 574
< Connection: keep-alive
< Content-Type: application/json
< Date: Thu, 13 Sep 2018 11:07:11 GMT
< Server: nginx
< X-Consul-Index: 23656
< X-Consul-Knownleader: true
< X-Consul-Lastcontact: 0
< X-Remote-Ip: 10.0.0.200
< X-Server-Time: Thursday, 13-Sep-2018 11:07:11 GMT
<
< {“Node”:{“ID”:"",“Node”:“master-stewie”,“Address”:“10.0.0.200”,“Datacenter”:“dc1”,“TaggedAddresses”:null,“Meta”:null,“CreateIndex”:16384,“ModifyIndex”:21235},“Services”:{“mysql:metrics-42002”:{“ID”:“mysql:metrics-42002”,“Service”:“mysql:metrics”,“Tags”:[“alias_master-stewie”,“scheme_https”],“Address”:"",“Port”:42002,“EnableTagOverride”:false,“CreateIndex”:16384,“ModifyIndex”:16384},“mysql:queries-0”:{“ID”:“mysql:queries-0”,“Service”:“mysql:queries”,“Tags”:[“alias_master-stewie”],“Address”:"",“Port”:0,“EnableTagOverride”:false,“CreateIndex”:21244,“ModifyIndex”:21244}}}
2018/09/13 19:53:51 request:

DELETE /prometheus/api/v1/series?match[]={job=“linux”,instance=“master-stewie”} HTTP/1.1
Host: 10.0.0.29
User-Agent: Go-http-client/1.1
Authorization: Basic
Accept-Encoding: gzip

2018/09/13 19:53:51 response:
< HTTP/1.1 500 Internal Server Error
< Access-Control-Allow-Headers: Accept, Authorization, Content-Type, Origin
< Access-Control-Allow-Methods: GET, OPTIONS
< Access-Control-Allow-Origin: *
< Access-Control-Expose-Headers: Date
< Connection: keep-alive
< Content-Type: application/json
< Date: Thu, 13 Sep 2018 11:07:11 GMT
< Server: nginx
<
< {“status”:“error”,“errorType”:“internal”,“error”:“not implemented”}
panic: interface conversion: interface {} is nil, not map[string]interface {}

goroutine 1 [running]:
github.com/percona/pmm-client/pmm.(*Admin).PurgeMetrics(0xc288c0, 0x7ffdf02f697b, 0xd, 0xc420176b28, 0xc42012db48, 0x81da62)
/tmp/go/src/github.com/percona/pmm-client/pmm/pmm.go:431 +0x5a4
main.glob…func43(0xc225a0, 0xc4200fcd60, 0x1, 0x2)
/tmp/go/src/github.com/percona/pmm-client/pmm-admin.go:1295 +0xd1
github.com/percona/pmm-client/vendor/github.com/spf13/cobra.(*Command).execute(0xc225a0, 0xc4200fcd20, 0x2, 0x2, 0xc225a0, 0xc4200fcd20)
/tmp/go/src/github.com/percona/pmm-client/vendor/github.com/spf13/cobra/command.go:654 +0x2a2
github.com/percona/pmm-client/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xc1cde0, 0xc42012ddb0, 0x49ba52, 0x0)
/tmp/go/src/github.com/percona/pmm-client/vendor/github.com/spf13/cobra/command.go:729 +0x2d4
github.com/percona/pmm-client/vendor/github.com/spf13/cobra.(*Command).Execute(0xc1cde0, 0xc460aa, 0x9524d8)
/tmp/go/src/github.com/percona/pmm-client/vendor/github.com/spf13/cobra/command.go:688 +0x2b
main.main()
/tmp/go/src/github.com/percona/pmm-client/pmm-admin.go:1563 +0x13b1

[andycutright]

Fan Doesn’t work in 1.14.1

Client side.

sudo pmm-admin info
pmm-admin 1.14.1

PMM Server | 10.0.0.29 (password-protected)
Client Name | master-stewie
Client Address | 10.0.0.200
Service Manager | linux-upstart

Go Version | 1.10.1
Runtime Info | linux/amd64

[Tim_Schmolzi]

I am seeing the same errors in the OVA version

[Michael_Coburn]

Hi Fan , andycutright , and Tim Schmolzi

Thanks for reporting this issue - turns out we’re trying to execute Prometheus 1.x (delete series) API calls against Prometheus 2.x backend. We’re tracking this issue in our JIRA under https://jira.percona.com/browse/PMM-2845 , and we are targeting our October release (tentatively Oct 10) for a fix.

https://prometheus.io/docs/prometheus/1.8/querying/api/#deleting-series
https://prometheus.io/docs/prometheus/latest/querying/api/#delete-series