leaderHealthCheck fails after update to 3.0.0

Percona Monitoring and Management (PMM) PMM 3.x
1

Hello!

After upgrading PMM HA to version 3.0.0, we encountered an issue where HAProxy leaderHealthCheck request receives a 401 code from pmm-server.

After a little investigation of the source code, I found that instead of a POST request to /v1/leaderHealthCheck, the server now expects a GET request to /v1/server/leaderHealthCheck.

However, when I check this endpoint using:

curl localhost/v1/server/leaderHealthCheck

I receive the following response:

{
  "error": "Internal server error.",
  "code": 13,
  "message": "Internal server error.",
  "details": []
}

In the Nginx logs, I see a 500 error code, and in the pmm-managed logs, the following error appears:

time="2025-02-20T14:21:10.761+00:00" level=info msg="Starting RPC /server.v1.ServerService/LeaderHealthCheck ..." request=eed10da5-ef95-11ef-8fcd-0242ac110002
time="2025-02-20T14:21:10.762+00:00" level=error msg="RPC /server.v1.ServerService/LeaderHealthCheck done in 62.967µs with panic: runtime error: invalid memory address or nil pointer dereference\nStack: goroutine 2249254 [running]:\nruntime/debug.Stack()\n\t/usr/local/go/src/runtime/debug/stack.go:26 +0x5e\ngithub.com/percona/pmm/managed/utils/interceptors.logRequest.func1()\n\t/home/builder/rpm/BUILD/pmm-d989d76c744b70da0a0ff9fbdc58010d3bc8c344/src/github.com/percona/pmm/managed/utils/interceptors/interceptors.go:48 +0x451\npanic({0x18a1ec0?, 0x2e60b00?})\n\t/usr/local/go/src/runtime/panic.go:785 +0x132\ngithub.com/percona/pmm/managed/services/server.(*Server).LeaderHealthCheck(0x2e6dac0?, {0x1afa200?, 0xc0009004e0?}, 0x8610795145695c7a?)\n\t/home/builder/rpm/BUILD/pmm-d989d76c744b70da0a0ff9fbdc58010d3bc8c344/src/github.com/percona/pmm/managed/services/server/server.go:232 +0x20\ngithub.com/percona/pmm/api/server/v1._ServerService_LeaderHealthCheck_Handler.func1({0x1f93118?, 0xc000febd10?}, {0x1a25b60?, 0xc000febb90?})\n\t/home/builder/rpm/BUILD/pmm-d989d76c744b70da0a0ff9fbdc58010d3bc8c344/src/github.com/percona/pmm/api/server/v1/server_grpc.pb.go:299 +0xcb\nmain.runGRPCServer.UnaryServerInterceptor.func7({0x1f93118, 0xc000febd10}, {0x1a25b60, 0xc000febb90}, 0x10?, 0xc000bc01e0)\n\t/home/builder/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.4.0/validator/validator.go:47 +0x65\nmain.runGRPCServer.ChainUnaryServer.func9.1({0x1f93118?, 0xc000febd10?}, {0x1a25b60?, 0xc000febb90?})\n\t/home/builder/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.4.0/chain.go:48 +0x45\nmain.runGRPCServer.UnaryServiceEnabledInterceptor.func6({0x1f93118, 0xc000febd10}, {0x1a25b60, 0xc000febb90}, 0xc00072e880, 0xc000fbe640)\n\t/home/builder/rpm/BUILD/pmm-d989d76c744b70da0a0ff9fbdc58010d3bc8c344/src/github.com/percona/pmm/managed/utils/interceptors/service.go:39 +0x8a\nmain.runGRPCServer.ChainUnaryServer.func9.1({0x1f93118?, 0xc000febd10?}, {0x1a25b60?, 0xc000febb90?})\n\t/home/builder/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.4.0/chain.go:48 +0x45\nmain.runGRPCServer.(*ServerMetrics).UnaryServerInterceptor.func4({0x1f93118, 0xc000febd10}, {0x1a25b60, 0xc000febb90}, 0x4?, 0xc000fbe680)\n\t/home/builder/go/pkg/mod/github.com/!percona-!lab/go-grpc-prometheus@v0.0.0-20230116133345-3487748d4592/server_metrics.go:114 +0x87\nmain.runGRPCServer.Unary.func5.1()\n\t/home/builder/rpm/BUILD/pmm-d989d76c744b70da0a0ff9fbdc58010d3bc8c344/src/github.com/percona/pmm/managed/utils/interceptors/interceptors.go:105 +0x58\ngithub.com/percona/pmm/managed/utils/interceptors.logRequest(0xc000fe6850, {0xc00129c150, 0x2e}, 0xc0008857d0)\n\t/home/builder/rpm/BUILD/pmm-d989d76c744b70da0a0ff9fbdc58010d3bc8c344/src/github.com/percona/pmm/managed/utils/interceptors/interceptors.go:77 +0x16b\nmain.runGRPCServer.Unary.func5({0x1f93118, 0xc000febb60}, {0x1a25b60, 0xc000febb90}, 0xc00072e880, 0xc000fbe680)\n\t/home/builder/rpm/BUILD/pmm-d989d76c744b70da0a0ff9fbdc58010d3bc8c344/src/github.com/percona/pmm/managed/utils/interceptors/interceptors.go:103 +0x2dd\nmain.runGRPCServer.ChainUnaryServer.func9({0x1f93118, 0xc000febb60}, {0x1a25b60, 0xc000febb90}, 0xc00072e880, 0x78?)\n\t/home/builder/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.4.0/chain.go:53 +0x123\ngithub.com/percona/pmm/api/server/v1._ServerService_LeaderHealthCheck_Handler({0x1afa200, 0xc000210d00}, {0x1f93118, 0xc000febb60}, 0xc000bf8300, 0xc000900630)\n\t/home/builder/rpm/BUILD/pmm-d989d76c744b70da0a0ff9fbdc58010d3bc8c344/src/github.com/percona/pmm/api/server/v1/server_grpc.pb.go:301 +0x143\ngoogle.golang.org/grpc.(*Server).processUnaryRPC(0xc00063e000, {0x1f93118, 0xc000febad0}, {0x1fa2f40, 0xc0001bd860}, 0xc000fa2a20, 0xc000900840, 0x2e91ad0, 0x0)\n\t/home/builder/go/pkg/mod/google.golang.org/grpc@v1.68.0/server.go:1394 +0xe2b\ngoogle.golang.org/grpc.(*Server).handleStream(0xc00063e000, {0x1fa2f40, 0xc0001bd860}, 0xc000fa2a20)\n\t/home/builder/go/pkg/mod/google.golang.org/grpc@v1.68.0/server.go:1805 +0xe8b\ngoogle.golang.org/grpc.(*Server).serveStreams.func2.1()\n\t/home/builder/go/pkg/mod/google.golang.org/grpc@v1.68.0/server.go:1029 +0x7f\ncreated by google.golang.org/grpc.(*Server).serveStreams.func2 in goroutine 368\n\t/home/builder/go/pkg/mod/google.golang.org/grpc@v1.68.0/server.go:1040 +0x125\n" request=eed10da5-ef95-11ef-8fcd-0242ac110002

Could you please advise on how to resolve this? Is there any additional configuration required after the upgrade?

Thanks in advance!

2

Hi, yes, we noticed this problem recently and fix is comming in 3.1.0

1 Like