Not the answer you need?
Register and ask your own question!

got error when execute pmm-admin purge

FanFan ContributorCurrent User Role Patron
via docker
1.13.0

[[email protected] fanboshi]# pmm-admin purge mysql:metrics 192.168.2.217_3307
panic: interface conversion: interface {} is nil, not map[string]interface {}

goroutine 1 [running]:
github.com/percona/pmm-client/pmm.(*Admin).PurgeMetrics(0xbe4e60, 0x7ffc1c4d28d6, 0xd, 0xc4202204a8, 0xc4201cbb68, 0x7ff262)
/tmp/go/src/github.com/percona/pmm-client/pmm/main.go:427 +0x5a4
main.glob..func39(0xbde460, 0xc42019d4c0, 0x2, 0x2)
/tmp/go/src/github.com/percona/pmm-client/pmm-admin.go:1179 +0xd1
github.com/percona/pmm-client/vendor/github.com/spf13/cobra.(*Command).execute(0xbde460, 0xc42019d440, 0x2, 0x2, 0xbde460, 0xc42019d440)
/tmp/go/src/github.com/percona/pmm-client/vendor/github.com/spf13/cobra/command.go:654 +0x2a2
github.com/percona/pmm-client/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xbdeb20, 0xc4201cbdd0, 0x49b302, 0x0)
/tmp/go/src/github.com/percona/pmm-client/vendor/github.com/spf13/cobra/command.go:729 +0x2d4
github.com/percona/pmm-client/vendor/github.com/spf13/cobra.(*Command).Execute(0xbdeb20, 0xc0263c, 0x92bf0f)
/tmp/go/src/github.com/percona/pmm-client/vendor/github.com/spf13/cobra/command.go:688 +0x2b
main.main()
/tmp/go/src/github.com/percona/pmm-client/pmm-admin.go:1398 +0x18dc

Comments

  • Michael CoburnMichael Coburn Principal Architect, Percona Percona Staff Role
    Hi Fan

    Can you run this with verbose option:
    pmm-admin --verbose ...
    
    Also, can you attach the logs:
    /var/log/pmm-mysql*.log
    
  • FanFan Contributor Current User Role Patron
    Hi Fan

    Can you run this with verbose option:
    pmm-admin --verbose ...
    
    Also, can you attach the logs:
    /var/log/pmm-mysql*.log
    

    this time i purge linux:metrics (don't support markdown, so the format is terrible)
    [[email protected] fanboshi]# pmm-admin add linux:metrics
    OK, now monitoring this system.
    [[email protected] fanboshi]# pmm-admin uninstall
    OK, 1 services were removed.
    [[email protected] fanboshi]# pmm-admin purge linux:metrics
    panic: interface conversion: interface {} is nil, not map[string]interface {}

    goroutine 1 [running]:
    github.com/percona/pmm-client/pmm.(*Admin).PurgeMetrics(0xbe4e60, 0x7ffe280718e9, 0xd, 0xc420220718, 0xc4201cbb68, 0x7ff262)
    /tmp/go/src/github.com/percona/pmm-client/pmm/main.go:427 +0x5a4
    main.glob..func39(0xbde460, 0xc4200e56b0, 0x1, 0x1)
    /tmp/go/src/github.com/percona/pmm-client/pmm-admin.go:1179 +0xd1
    github.com/percona/pmm-client/vendor/github.com/spf13/cobra.(*Command).execute(0xbde460, 0xc4200e5660, 0x1, 0x1, 0xbde460, 0xc4200e5660)
    /tmp/go/src/github.com/percona/pmm-client/vendor/github.com/spf13/cobra/command.go:654 +0x2a2
    github.com/percona/pmm-client/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xbdeb20, 0xc4201cbdd0, 0x49b302, 0x0)
    /tmp/go/src/github.com/percona/pmm-client/vendor/github.com/spf13/cobra/command.go:729 +0x2d4
    github.com/percona/pmm-client/vendor/github.com/spf13/cobra.(*Command).Execute(0xbdeb20, 0xc0263c, 0x92bf0f)
    /tmp/go/src/github.com/percona/pmm-client/vendor/github.com/spf13/cobra/command.go:688 +0x2b
    main.main()
    /tmp/go/src/github.com/percona/pmm-client/pmm-admin.go:1398 +0x18dc
    [[email protected] fanboshi]# pmm-admin --verbose purge linux:metrics
    2018/08/04 12:51:18 request:
    > GET /qan-api/ping HTTP/1.1
    > Host: 10.1.2.143:8080
    > User-Agent: Go-http-client/1.1
    > Authorization: Basic bWZ3X2RiYTpacWRSSGV2bmE4
    > Accept-Encoding: gzip
    >
    >
    2018/08/04 12:51:18 response:
    < HTTP/1.1 200 OK
    < Access-Control-Allow-Headers: Content-Type,Authorization
    < Access-Control-Allow-Methods: GET,PUT,POST,DELETE
    < Access-Control-Allow-Origin: *
    < Cache-Control: no-store
    < Connection: keep-alive
    < Content-Type: text/plain; charset=utf-8
    < Date: Sat, 04 Aug 2018 04:51:18 GMT
    < Pragma: no-cache
    < Server: nginx
    < Strict-Transport-Security: max-age=63072000; includeSubDomains
    < X-Content-Type-Options: nosniff
    < X-Frame-Options: DENY
    < X-Percona-Qan-Api-Version: 1.13.0-1807241501.89a7875
    < X-Xss-Protection: 1; mode=block
    < Content-Length: 0
    <
    <
    2018/08/04 12:51:18 request:
    > GET /v1/status/leader HTTP/1.1
    > Host: 10.1.2.143:8080
    > User-Agent: Go-http-client/1.1
    > Authorization: Basic bWZ3X2RiYTpacWRSSGV2bmE4
    > Accept-Encoding: gzip
    >
    >
    2018/08/04 12:51:18 response:
    < HTTP/1.1 200 OK
    < Content-Length: 16
    < Connection: keep-alive
    < Content-Type: application/json
    < Date: Sat, 04 Aug 2018 04:51:18 GMT
    < Server: nginx
    < X-Remote-Ip: 10.1.2.142
    < X-Server-Time: Saturday, 04-Aug-2018 04:51:18 GMT
    <
    < "127.0.0.1:8300"
    2018/08/04 12:51:18 request:
    > GET /qan-api/ping HTTP/1.1
    > Host: 10.1.2.143:8080
    > User-Agent: Go-http-client/1.1
    > Accept-Encoding: gzip
    >
    >
    2018/08/04 12:51:18 response:
    < HTTP/1.1 401 Unauthorized
    < Content-Length: 188
    < Connection: keep-alive
    < Content-Type: text/html
    < Date: Sat, 04 Aug 2018 04:51:18 GMT
    < Server: nginx
    < Www-Authenticate: Basic realm="on"
    <
    < <html>
    < <head><title>401 Authorization Required</title></head>
    < <body bgcolor="white">
    < <center><h1>401 Authorization Required</h1></center>
    < <hr><center>nginx</center>
    < </body>
    < </html>
    <
    2018/08/04 12:51:18 request:
    > GET /v1/catalog/node/192.168.2.142 HTTP/1.1
    > Host: 10.1.2.143:8080
    > User-Agent: Go-http-client/1.1
    > Authorization: Basic bWZ3X2RiYTpacWRSSGV2bmE4
    > Accept-Encoding: gzip
    >
    >
    2018/08/04 12:51:18 response:
    < HTTP/1.1 200 OK
    < Content-Length: 170
    < Connection: keep-alive
    < Content-Type: application/json
    < Date: Sat, 04 Aug 2018 04:51:18 GMT
    < Server: nginx
    < X-Consul-Index: 4680
    < X-Consul-Knownleader: true
    < X-Consul-Lastcontact: 0
    < X-Remote-Ip: 10.1.2.142
    < X-Server-Time: Saturday, 04-Aug-2018 04:51:18 GMT
    <
    < {"Node":{"ID":"","Node":"192.168.2.142","Address":"10.1.2.142","Datacenter":"dc1","TaggedAddresses":null,"Meta":null,"CreateIndex":4678,"ModifyIndex":4678},"Services":{}}
    2018/08/04 12:51:18 request:
    > DELETE /prometheus/api/v1/series?match[]={job="linux",instance="192.168.2.142"} HTTP/1.1
    > Host: 10.1.2.143:8080
    > User-Agent: Go-http-client/1.1
    > Authorization: Basic bWZ3X2RiYTpacWRSSGV2bmE4
    > Accept-Encoding: gzip
    >
    >
    2018/08/04 12:51:18 response:
    < HTTP/1.1 500 Internal Server Error
    < Access-Control-Allow-Headers: Accept, Authorization, Content-Type, Origin
    < Access-Control-Allow-Methods: GET, OPTIONS
    < Access-Control-Allow-Origin: *
    < Access-Control-Expose-Headers: Date
    < Connection: keep-alive
    < Content-Type: application/json
    < Date: Sat, 04 Aug 2018 04:51:18 GMT
    < Server: nginx
    <
    < {"status":"error","errorType":"internal","error":"not implemented"}
    panic: interface conversion: interface {} is nil, not map[string]interface {}

    goroutine 1 [running]:
    github.com/percona/pmm-client/pmm.(*Admin).PurgeMetrics(0xbe4e60, 0x7ffc91ae58e9, 0xd, 0xc420220648, 0xc4201cbb68, 0x7ff262)
    /tmp/go/src/github.com/percona/pmm-client/pmm/main.go:427 +0x5a4
    main.glob..func39(0xbde460, 0xc42019d480, 0x1, 0x2)
    /tmp/go/src/github.com/percona/pmm-client/pmm-admin.go:1179 +0xd1
    github.com/percona/pmm-client/vendor/github.com/spf13/cobra.(*Command).execute(0xbde460, 0xc42019d440, 0x2, 0x2, 0xbde460, 0xc42019d440)
    /tmp/go/src/github.com/percona/pmm-client/vendor/github.com/spf13/cobra/command.go:654 +0x2a2
    github.com/percona/pmm-client/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xbdeb20, 0xc4201cbdd0, 0x49b302, 0x0)
    /tmp/go/src/github.com/percona/pmm-client/vendor/github.com/spf13/cobra/command.go:729 +0x2d4
    github.com/percona/pmm-client/vendor/github.com/spf13/cobra.(*Command).Execute(0xbdeb20, 0xc0263c, 0x92bf0f)
    /tmp/go/src/github.com/percona/pmm-client/vendor/github.com/spf13/cobra/command.go:688 +0x2b
    main.main()
    /tmp/go/src/github.com/percona/pmm-client/pmm-admin.go:1398 +0x18dc



    log
    [[email protected] log]# less /var/log/pmm-linux-metrics-42000.log
    time="2018-08-04T12:53:21+08:00" level=info msg="Starting node_exporter (version=, branch=, revision=)" source="node_exporter.go:142"
    time="2018-08-04T12:53:21+08:00" level=info msg="Build context (go=go1.10.1, user=, date=)" source="node_exporter.go:143"
    time="2018-08-04T12:53:21+08:00" level=info msg="Enabled collectors:" source="node_exporter.go:162"
    time="2018-08-04T12:53:21+08:00" level=info msg=" - diskstats" source="node_exporter.go:164"
    time="2018-08-04T12:53:21+08:00" level=info msg=" - filefd" source="node_exporter.go:164"
    time="2018-08-04T12:53:21+08:00" level=info msg=" - filesystem" source="node_exporter.go:164"
    time="2018-08-04T12:53:21+08:00" level=info msg=" - meminfo" source="node_exporter.go:164"
    time="2018-08-04T12:53:21+08:00" level=info msg=" - netdev" source="node_exporter.go:164"
    time="2018-08-04T12:53:21+08:00" level=info msg=" - uname" source="node_exporter.go:164"
    time="2018-08-04T12:53:21+08:00" level=info msg=" - meminfo_numa" source="node_exporter.go:164"
    time="2018-08-04T12:53:21+08:00" level=info msg=" - loadavg" source="node_exporter.go:164"
    time="2018-08-04T12:53:21+08:00" level=info msg=" - netstat" source="node_exporter.go:164"
    time="2018-08-04T12:53:21+08:00" level=info msg=" - stat" source="node_exporter.go:164"
    time="2018-08-04T12:53:21+08:00" level=info msg=" - time" source="node_exporter.go:164"
    time="2018-08-04T12:53:21+08:00" level=info msg=" - vmstat" source="node_exporter.go:164"
    time="2018-08-04T12:53:21+08:00" level=info msg="HTTP Basic authentication is enabled." source="basic_auth.go:105"
    time="2018-08-04T12:53:21+08:00" level=info msg="Starting HTTPS server for https://10.1.2.142:42000/metrics ..." source="server.go:106"
  • FanFan Contributor Current User Role Patron
    Hi Fan

    Can you run this with verbose option:
    pmm-admin --verbose ...
    
    Also, can you attach the logs:
    /var/log/pmm-mysql*.log
    

    so? Is it a bug or something?
  • FanFan Contributor Current User Role Patron
    Has this bug been fixed in 1.14? @Michael Coburn
  • andycutrightandycutright Entrant Current User Role Beginner
    https://www.percona.com/forums/member/22577-michael-coburn

    I see the same issue. I'm running the OVA and MySQL community server on the the client. All the pmm services are running properly on both boxes, as far as I can tell. I removed the linux:metric service, and then purged. I don't see a log file in /var/log for the linux:metrics service.


    sudo pmm-admin --verbose purge linux:metrics
    # Some data has been elided for security
    sudo pmm-admin --verbose purge linux:metrics
    2018/09/13 19:53:51 request:
    2018/09/13 19:53:51 request:
    > GET /qan-api/ping HTTP/1.1
    > Host: 10.0.0.29
    > User-Agent: Go-http-client/1.1
    > Authorization: Basic ELIDED
    > Accept-Encoding: gzip
    >
    >
    2018/09/13 19:53:51 response:
    < HTTP/1.1 200 OK
    < Access-Control-Allow-Headers: Content-Type,Authorization
    < Access-Control-Allow-Methods: GET,PUT,POST,DELETE
    < Access-Control-Allow-Origin: *
    < Cache-Control: no-store
    < Connection: keep-alive
    < Content-Type: text/plain; charset=utf-8
    < Date: Thu, 13 Sep 2018 11:07:11 GMT
    < Pragma: no-cache
    < Server: nginx
    < Strict-Transport-Security: max-age=63072000; includeSubDomains
    < X-Content-Type-Options: nosniff
    < X-Frame-Options: DENY
    < X-Percona-Qan-Api-Version: 1.14.1-1809060744.89a7875
    < X-Xss-Protection: 1; mode=block
    < Content-Length: 0
    <
    <
    2018/09/13 19:53:51 request:
    > GET /v1/status/leader HTTP/1.1
    > Host: 10.0.0.29
    > User-Agent: Go-http-client/1.1
    > Authorization: Basic
    > Accept-Encoding: gzip
    >
    >
    2018/09/13 19:53:51 response:
    < HTTP/1.1 200 OK
    < Content-Length: 16
    < Connection: keep-alive
    < Content-Type: application/json
    < Date: Thu, 13 Sep 2018 11:07:11 GMT
    < Server: nginx
    < X-Remote-Ip: 10.0.0.200
    < X-Server-Time: Thursday, 13-Sep-2018 11:07:11 GMT
    <
    < "127.0.0.1:8300"
    2018/09/13 19:53:51 request:
    > GET /qan-api/ping HTTP/1.1
    > Host: 10.0.0.29
    > User-Agent: Go-http-client/1.1
    > Accept-Encoding: gzip
    >
    >
    2018/09/13 19:53:51 response:
    < HTTP/1.1 401 Unauthorized
    < Content-Length: 188
    < Connection: keep-alive
    < Content-Type: text/html
    < Date: Thu, 13 Sep 2018 11:07:11 GMT
    < Server: nginx
    < Www-Authenticate: Basic realm="on"
    <
    < <html>
    < <head><title>401 Authorization Required</title></head>
    < <body bgcolor="white">
    < <center><h1>401 Authorization Required</h1></center>
    < <hr><center>nginx</center>
    < </body>
    < </html>
    <
    2018/09/13 19:53:51 request:
    > GET /v1/catalog/node/master-stewie HTTP/1.1
    > Host: 10.0.0.29
    > User-Agent: Go-http-client/1.1
    > Authorization: Basic
    > Accept-Encoding: gzip
    >
    >
    2018/09/13 19:53:51 response:
    < HTTP/1.1 200 OK
    < Content-Length: 574
    < Connection: keep-alive
    < Content-Type: application/json
    < Date: Thu, 13 Sep 2018 11:07:11 GMT
    < Server: nginx
    < X-Consul-Index: 23656
    < X-Consul-Knownleader: true
    < X-Consul-Lastcontact: 0
    < X-Remote-Ip: 10.0.0.200
    < X-Server-Time: Thursday, 13-Sep-2018 11:07:11 GMT
    <
    < {"Node":{"ID":"","Node":"master-stewie","Address":"10.0.0.200","Datacenter":"dc1","TaggedAddresses":null,"Meta":null,"CreateIndex":16384,"ModifyIndex":21235},"Services":{"mysql:metrics-42002":{"ID":"mysql:metrics-42002","Service":"mysql:metrics","Tags":["alias_master-stewie","scheme_https"],"Address":"","Port":42002,"EnableTagOverride":false,"CreateIndex":16384,"ModifyIndex":16384},"mysql:queries-0":{"ID":"mysql:queries-0","Service":"mysql:queries","Tags":["alias_master-stewie"],"Address":"","Port":0,"EnableTagOverride":false,"CreateIndex":21244,"ModifyIndex":21244}}}
    2018/09/13 19:53:51 request:
    > DELETE /prometheus/api/v1/series?match[]={job="linux",instance="master-stewie"} HTTP/1.1
    > Host: 10.0.0.29
    > User-Agent: Go-http-client/1.1
    > Authorization: Basic
    > Accept-Encoding: gzip
    >
    >
    2018/09/13 19:53:51 response:
    < HTTP/1.1 500 Internal Server Error
    < Access-Control-Allow-Headers: Accept, Authorization, Content-Type, Origin
    < Access-Control-Allow-Methods: GET, OPTIONS
    < Access-Control-Allow-Origin: *
    < Access-Control-Expose-Headers: Date
    < Connection: keep-alive
    < Content-Type: application/json
    < Date: Thu, 13 Sep 2018 11:07:11 GMT
    < Server: nginx
    <
    < {"status":"error","errorType":"internal","error":"not implemented"}
    panic: interface conversion: interface {} is nil, not map[string]interface {}

    goroutine 1 [running]:
    github.com/percona/pmm-client/pmm.(*Admin).PurgeMetrics(0xc288c0, 0x7ffdf02f697b, 0xd, 0xc420176b28, 0xc42012db48, 0x81da62)
    /tmp/go/src/github.com/percona/pmm-client/pmm/pmm.go:431 +0x5a4
    main.glob..func43(0xc225a0, 0xc4200fcd60, 0x1, 0x2)
    /tmp/go/src/github.com/percona/pmm-client/pmm-admin.go:1295 +0xd1
    github.com/percona/pmm-client/vendor/github.com/spf13/cobra.(*Command).execute(0xc225a0, 0xc4200fcd20, 0x2, 0x2, 0xc225a0, 0xc4200fcd20)
    /tmp/go/src/github.com/percona/pmm-client/vendor/github.com/spf13/cobra/command.go:654 +0x2a2
    github.com/percona/pmm-client/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xc1cde0, 0xc42012ddb0, 0x49ba52, 0x0)
    /tmp/go/src/github.com/percona/pmm-client/vendor/github.com/spf13/cobra/command.go:729 +0x2d4
    github.com/percona/pmm-client/vendor/github.com/spf13/cobra.(*Command).Execute(0xc1cde0, 0xc460aa, 0x9524d8)
    /tmp/go/src/github.com/percona/pmm-client/vendor/github.com/spf13/cobra/command.go:688 +0x2b
    main.main()
    /tmp/go/src/github.com/percona/pmm-client/pmm-admin.go:1563 +0x13b1
  • andycutrightandycutright Entrant Current User Role Beginner
    Fan Doesn't work in 1.14.1

    Client side.

    sudo pmm-admin info
    pmm-admin 1.14.1

    PMM Server | 10.0.0.29 (password-protected)
    Client Name | master-stewie
    Client Address | 10.0.0.200
    Service Manager | linux-upstart

    Go Version | 1.10.1
    Runtime Info | linux/amd64
  • Tim SchmolziTim Schmolzi Entrant Inactive User Role Beginner
    I am seeing the same errors in the OVA version
  • Michael CoburnMichael Coburn Principal Architect, Percona Percona Staff Role
    Hi Fan , andycutright , and Tim Schmolzi

    Thanks for reporting this issue - turns out we're trying to execute Prometheus 1.x (delete series) API calls against Prometheus 2.x backend. We're tracking this issue in our JIRA under https://jira.percona.com/browse/PMM-2845 , and we are targeting our October release (tentatively Oct 10) for a fix.

    https://prometheus.io/docs/prometheus/1.8/querying/api/#deleting-series
    https://prometheus.io/docs/prometheus/latest/querying/api/#delete-series
Sign In or Register to comment.

MySQL, InnoDB, MariaDB and MongoDB are trademarks of their respective owners.
Copyright ©2005 - 2020 Percona LLC. All rights reserved.