I set up a PMM on 3.2.0 version on our production environment. Today I saw that upgrade to 3.3.0 is ready, so I want to try upgrade pmm server by using upgrade function from UI. Unfortunately, I got “Internal server error” message while trying to start upgrade:
but there is an error code in response - error code 13
{
“error”: “Internal server error.”,
“code”: 13,
“message”: “Internal server error.”,
“details”:
}
I tried to check logs on pmm-server docker container but I don not find any useful information about causes. Do you have any suggestions? What should I check to find the problem?
Sorry for the late answer - you’re right, somehow I missed pmm-managed.log I suppose. When I’m clicking upgrade in logs I’m getting an error
”Auth method is not service account token”
Logs below:
time=“2025-07-31T18:17:55.623+00:00” level=info msg=“Starting RPC /server.v1.ServerService/StartUpdate …” request=ae14b331-6e3a-11f0-ab16-f6f66f1797fc time=“2025-07-31T18:17:55.623+00:00” level=info msg=“Using default Percona Platform address "``https://check.percona.com``".” time=“2025-07-31T18:17:56.060+00:00” level=warning msg=“No explicit rule, falling back to Grafana admin.” component=grafana/auth req=“GET /v1/update?hostname=4c0dbbd9e505&newImageName=perconalab%2Fpmm-server%3A3.3.1&stopWatchtower=false” time=“2025-07-31T18:17:56.061+00:00” level=warning msg=“clientError: GET ``http://127.0.0.1:3000/api/auth/serviceaccount`` → 400 {"message":"Auth method is not service account token"}” component=grafana/auth req=“GET /v1/update?hostname=4c0dbbd9e505&newImageName=perconalab%2Fpmm-server%3A3.3.1&stopWatchtower=false” time=“2025-07-31T18:17:56.061+00:00” level=warning msg=“No explicit rule, falling back to Grafana admin.” component=grafana/auth req=“GET /v1/update?hostname=4c0dbbd9e505&newImageName=perconalab%2Fpmm-server%3A3.3.1&stopWatchtower=false” time=“2025-07-31T18:17:56.062+00:00” level=warning msg=“clientError: GET ``http://127.0.0.1:3000/api/auth/serviceaccount`` → 400 {"message":"Auth method is not service account token"}” component=grafana/auth req=“GET /v1/update?hostname=4c0dbbd9e505&newImageName=perconalab%2Fpmm-server%3A3.3.1&stopWatchtower=false” time=“2025-07-31T18:17:56.062+00:00” level=error msg=“Failed to trigger update” error=“received non-OK response: 401” service=updater time=“2025-07-31T18:17:56.062+00:00” level=error msg=“RPC /server.v1.ServerService/StartUpdate done in 439.065931ms with unexpected error: received non-OK response: 401\``ngithub.com/percona/pmm/managed/services/server.(*Updater).sendRequestToWatchtower\n\t/home/builder/rpm/BUILD/pmm-60c952647f6476f651083dbd9b47f02343636998/src/github.com/percona/pmm/managed/services/server/updater.go:153\ngithub.com/percona/pmm/managed/services/server.(*Updater).StartUpdate\n\t/home/builder/rpm/BUILD/pmm-60c952647f6476f651083dbd9b47f02343636998/src/github.com/percona/pmm/managed/services/server/updater.go:217\ngithub.com/percona/pmm/managed/services/server.(*Server).StartUpdate\n\t/home/builder/rpm/BUILD/pmm-60c952647f6476f651083dbd9b47f02343636998/src/github.com/percona/pmm/managed/services/server/server.go:362\ngithub.com/percona/pmm/api/server/v1._ServerService_StartUpdate_Handler.func1\n\t/home/builder/rpm/BUILD/pmm-60c952647f6476f651083dbd9b47f02343636998/src/github.com/percona/pmm/api/server/v1/server_grpc.pb.go:372\nmain.runGRPCServer.UnaryServerInterceptor.func7\n\t/home/builder/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.4.0/validator/validator.go:47\nmain.runGRPCServer.ChainUnaryServer.func9.1\n\t/home/builder/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.4.0/chain.go:48\nmain.runGRPCServer.UnaryServiceEnabledInterceptor.func6\n\t/home/builder/rpm/BUILD/pmm-60c952647f6476f651083dbd9b47f02343636998/src/github.com/percona/pmm/managed/utils/interceptors/service.go:39\nmain.runGRPCServer.ChainUnaryServer.func9.1\n\t/home/builder/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.4.0/chain.go:48\nmain.runGRPCServer.(*ServerMetrics).UnaryServerInterceptor.func4\n\t/home/builder/go/pkg/mod/github.com/!percona-!lab/go-grpc-prometheus@v0.0.0-20230116133345-3487748d4592/server_metrics.go:114\nmain.runGRPCServer.Unary.func5.1\n\t/home/builder/rpm/BUILD/pmm-60c952647f6476f651083dbd9b47f02343636998/src/github.com/percona/pmm/managed/utils/interceptors/interceptors.go:105\ngithub.com/percona/pmm/managed/utils/interceptors.logRequest\n\t/home/builder/rpm/BUILD/pmm-60c952647f6476f651083dbd9b47f02343636998/src/github.com/percona/pmm/managed/utils/interceptors/interceptors.go:77\nmain.runGRPCServer.Unary.func5\n\t/home/builder/rpm/BUILD/pmm-60c952647f6476f651083dbd9b47f02343636998/src/github.com/percona/pmm/managed/utils/interceptors/interceptors.go:103\nmain.runGRPCServer.ChainUnaryServer.func9\n\t/home/builder/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.4.0/chain.go:53\ngithub.com/percona/pmm/api/server/v1._ServerService_StartUpdate_Handler\n\t/home/builder/rpm/BUILD/pmm-60c952647f6476f651083dbd9b47f02343636998/src/github.com/percona/pmm/api/server/v1/server_grpc.pb.go:374\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/home/builder/go/pkg/mod/google.golang.org/grpc@v1.72.0/server.go:1405\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/home/builder/go/pkg/mod/google.golang.org/grpc@v1.72.0/server.go:1815\ngoogle.golang.org/grpc.(*Server).serveStreams.func2.1\n\t/home/builder/go/pkg/mod/google.golang.org/grpc@v1.72.0/server.go:1035\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1700”`` request=ae14b331-6e3a-11f0-ab16-f6f66f1797fc time=“2025-07-31T18:17:56.261+00:00” level=info msg=“Starting RPC /server.v1.ServerService/ListChangeLogs …” request=ae760d0c-6e3a-11f0-ab16-f6f66f1797fc time=“2025-07-31T18:17:56.261+00:00” level=info msg=“Using default Percona Platform address "``https://check.percona.com``".” time=“2025-07-31T18:17:56.391+00:00” level=info msg=“RPC /server.v1.ServerService/ListChangeLogs done in 130.319345ms.” request=ae760d0c-6e3a-11f0-ab16-f6f66f1797fc
