I have a problem with pmm-agent, I’m trying to connect to the pmm server, but it returns the error:
time=“2025-01-16T12:47:32.023+00:00” level=info msg=“Connecting to https://pmm-client:***@pmm..com.br:443/ …” component=client
time=“2025-01-16T12:47:32.034+00:00” level=info msg=“Connected to pmm..com.br:443.” component=client
time=“2025-01-16T12:47:32.034+00:00” level=info msg=“Establishing two-way communication channel …” component=client
time=“2025-01-16T12:47:32.034+00:00” level=debug msg=“Sending message (4 bytes): id:1 ping:{}.” component=channel
time=“2025-01-16T12:47:32.045+00:00” level=debug msg=“Closing with error: rpc error: code = PermissionDenied desc = unexpected HTTP status code received from server: 403 (Forbidden); transport: received unexpected content-type "text/html"\nfailed to receive message\ngithub.com/percona/pmm/agent/client/channel.(*Channel).runReceiver\n\t/tmp/go/src/github.com/percona/pmm/agent/client/channel/channel.go :220\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1700” component=channel
time=“2025-01-16T12:47:32.045+00:00” level=debug msg=“Exiting receiver goroutine.” component=channel
time=“2025-01-16T12:47:32.045+00:00” level=error msg=“Failed to establish two-way communication channel: unexpected HTTP status code received from server: 403 (Forbidden); transport: received unexpected content -type "text/html".” component=client
time=“2025-01-16T12:47:32.045+00:00” level=debug msg=“Connection closed.” component=client
I have already checked the access credentials and they are correct
Hi, please show us which command exactly you are running that produces this error.
pmm-admin config --server-url=https://pmm-client:“pw”@pmm.“server”.com.br/ --server-insecure-tls
Just replace the password and domain
are you using any proxies on the client side? check http_proxy, https_proxy, HTTP_PROXY and HTTPS_PROXY variables to make sure that the client is connecting to the proper place
When checking the agent status, the following appears
pmm-admin status
Agent ID : /agent_id/241324c2-7186-447b-8fb9-abc67304247d
Node ID :
Node name:
PMM Server:
URL : https://pmm.“server”.com.br:443/
Version:
PMM Client:
Connected : false
Connection uptime: 0
pmm-admin version: 2.44.0
pmm-agent version: 2.44.0
Agents:
@ALFR3D022,
pmm-admin config --server-url=https://pmm-client:“pw”@pmm.“server”.com.br/ --server-insecure-tls
Do not use " marks in the URL for the password. The quote marks will be considered as part of the password. The error’s above you showed have 403 Forbidden, which means authentication is failing. Did you create the user ‘pmm-client’ inside the PMM UI?
I put the quotation marks in between just as an example, in fact the password is correct, like this:
pmm-admin config --server-url=https://pmm-client:password@pmm.server.com.br/ --server-insecure-tls
Yes, I created the pmm-client user in the PMM UI with administrator access
Try a simple API. Does this work?
curl https://pmm-client:password@pmm.server.br/v1/user/list ? If that doesn’t work, then your UN/PW are incorrect.
curl https://pmm-client:password@pmm.server.com.br/v1/user/list
curl: (77) error setting certificate file: /etc/pki/tls/certs/ca-bundle.crt
curl https://pmm-client:password@pmm.server.com.br/v1/user/list -k
{
“error”: “Method Not Allowed”,
“code”: 12,
“message”: “Method Not Allowed”
}
I’m using --server-insecure-tls
Do some debugging and see what is failing.
pmm-admin config --trace --debug --server-url=https://pmm-client:password@pmm.server.com.br/ --server-insecure-tls
time=“2025-01-16T15:13:43.167+00:00” level=debug msg=“Called delete old events” func=“connectionuptime.(*Service).RunCleanupGoroutine.func1” file=“connectionuptime/service.go:108” component=connection-uptime-service
time=“2025-01-16T15:13:45.578+00:00” level=info msg=“Connecting to https://pmm-client:***@pmm.server.com.br:443/ …” func=client.dial file=“client/client.go:762” component=client
time=“2025-01-16T15:13:45.578+00:00” level=trace msg=“[core] original dial target is: "pmm.server.com.br:443"” func=“config.(*gRPCLogger).Infoln” file=“config/logger.go:37” component=grpclog
time=“2025-01-16T15:13:45.578+00:00” level=trace msg=“[core] [Channel #280]Channel created” func=“config.(*gRPCLogger).Infoln” file=“config/logger.go:37” component=grpclog
time=“2025-01-16T15:13:45.578+00:00” level=trace msg=“[core] [Channel #280]parsed dial target is: resolver.Target{URL:url.URL{Scheme:"passthrough", Opaque:"", User:(*url.Userinfo)(nil), Host:"", Path:"/pmm.server.com.br:443", RawPath:"", OmitHost:false, ForceQuery:false, RawQuery:"", Fragment:"", RawFragment:""}}” func=“config.(*gRPCLogger).Infoln” file=“config/logger.go:37” component=grpclog
time=“2025-01-16T15:13:45.578+00:00” level=trace msg=“[core] [Channel #280]Channel authority set to "pmm.server.com.br"” func=“config.(*gRPCLogger).Infoln” file=“config/logger.go:37” component=grpclog
time=“2025-01-16T15:13:45.578+00:00” level=trace msg=“[core] [Channel #280]Resolver state updated: {\n "Addresses": [\n {\n "Addr": "pmm.server.com.br:443",\n "ServerName": "",\n "Attributes": null,\n "BalancerAttributes": null,\n "Metadata": null\n }\n ],\n "Endpoints": [\n {\n "Addresses": [\n {\n "Addr": "pmm.server.com.br:443",\n "ServerName": "",\n "Attributes": null,\n "BalancerAttributes": null,\n "Metadata": null\n }\n ],\n "Attributes": null\n }\n ],\n "ServiceConfig": null,\n "Attributes": null\n} (resolver returned new addresses)” func=“config.(*gRPCLogger).Infoln” file=“config/logger.go:37” component=grpclog
time=“2025-01-16T15:13:45.578+00:00” level=trace msg=“[core] [Channel #280]Channel switches to new LB policy "pick_first"” func=“config.(*gRPCLogger).Infoln” file=“config/logger.go:37” component=grpclog
time=“2025-01-16T15:13:45.578+00:00” level=trace msg=“[pick-first-lb] [pick-first-lb 0xc000a04270] Received new config {\n "shuffleAddressList": false\n}, resolver state {\n "Addresses": [\n {\n "Addr": "pmm.server.com.br:443",\n "ServerName": "",\n "Attributes": null,\n "BalancerAttributes": null,\n "Metadata": null\n }\n ],\n "Endpoints": [\n {\n "Addresses": [\n {\n "Addr": "pmm.server.com.br:443",\n "ServerName": "",\n "Attributes": null,\n "BalancerAttributes": null,\n "Metadata": null\n }\n ],\n "Attributes": null\n }\n ],\n "ServiceConfig": null,\n "Attributes": null\n}” func=“config.(*gRPCLogger).Infoln” file=“config/logger.go:37” component=grpclog
time=“2025-01-16T15:13:45.578+00:00” level=trace msg=“[core] [Channel #280 SubChannel #281]Subchannel created” func=“config.(*gRPCLogger).Infoln” file=“config/logger.go:37” component=grpclog
time=“2025-01-16T15:13:45.578+00:00” level=trace msg=“[core] [Channel #280]Channel Connectivity change to CONNECTING” func=“config.(*gRPCLogger).Infoln” file=“config/logger.go:37” component=grpclog
time=“2025-01-16T15:13:45.578+00:00” level=trace msg=“[core] [Channel #280]Channel exiting idle mode” func=“config.(*gRPCLogger).Infoln” file=“config/logger.go:37” component=grpclog
time=“2025-01-16T15:13:45.579+00:00” level=trace msg=“[core] [Channel #280 SubChannel #281]Subchannel Connectivity change to CONNECTING” func=“config.(*gRPCLogger).Infoln” file=“config/logger.go:37” component=grpclog
time=“2025-01-16T15:13:45.579+00:00” level=trace msg=“[core] [Channel #280 SubChannel #281]Subchannel picks a new address "pmm.server.com.br:443" to connect” func=“config.(*gRPCLogger).Infoln” file=“config/logger.go:37” component=grpclog
time=“2025-01-16T15:13:45.579+00:00” level=trace msg=“[pick-first-lb] [pick-first-lb 0xc000a04270] Received SubConn state update: 0xc000a04360, {ConnectivityState:CONNECTING ConnectionError: connectedAddress:{Addr: ServerName: Attributes: BalancerAttributes: Metadata:}}” func=“config.(*gRPCLogger).Infoln” file=“config/logger.go:37” component=grpclog
time=“2025-01-16T15:13:45.600+00:00” level=trace msg=“[core] [Channel #280 SubChannel #281]Subchannel Connectivity change to READY” func=“config.(*gRPCLogger).Infoln” file=“config/logger.go:37” component=grpclog
time=“2025-01-16T15:13:45.600+00:00” level=trace msg=“[pick-first-lb] [pick-first-lb 0xc000a04270] Received SubConn state update: 0xc000a04360, {ConnectivityState:READY ConnectionError: connectedAddress:{Addr:pmm.server.com.br:443 ServerName:pmm.server.com.br Attributes: BalancerAttributes: Metadata:}}” func=“config.(*gRPCLogger).Infoln” file=“config/logger.go:37” component=grpclog
time=“2025-01-16T15:13:45.600+00:00” level=trace msg=“[core] [Channel #280]Channel Connectivity change to READY” func=“config.(*gRPCLogger).Infoln” file=“config/logger.go:37” component=grpclog
time=“2025-01-16T15:13:45.601+00:00” level=info msg=“Connected to pmm.server.com.br:443.” func=client.dial file=“client/client.go:775” component=client
time=“2025-01-16T15:13:45.601+00:00” level=info msg=“Establishing two-way communication channel …” func=client.dial file=“client/client.go:796” component=client
time=“2025-01-16T15:13:45.601+00:00” level=debug msg=“Sending message (4 bytes): id:1 ping:{}.” func=“channel.(*Channel).send” file=“channel/channel.go:195” component=channel
time=“2025-01-16T15:13:45.617+00:00” level=debug msg=“Closing with error: rpc error: code = PermissionDenied desc = unexpected HTTP status code received from server: 403 (Forbidden); transport: received unexpected content-type "text/html"\nfailed to receive message\ngithub.com/percona/pmm/agent/client/channel.(*Channel).runReceiver\n\t/tmp/go/src/github.com/percona/pmm/agent/client/channel/channel.go:220\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1700” func=“channel.(*Channel).close.func1” file=“channel/channel.go:120” component=channel
time=“2025-01-16T15:13:45.617+00:00” level=debug msg=“Exiting receiver goroutine.” func=“channel.(*Channel).runReceiver.func1” file=“channel/channel.go:214” component=channel
time=“2025-01-16T15:13:45.617+00:00” level=error msg=“Failed to establish two-way communication channel: unexpected HTTP status code received from server: 403 (Forbidden); transport: received unexpected content-type "text/html".” func=client.dial file=“client/client.go:823” component=client
time=“2025-01-16T15:13:45.617+00:00” level=trace msg=“[core] [Channel #280]Channel Connectivity change to SHUTDOWN” func=“config.(*gRPCLogger).Infoln” file=“config/logger.go:37” component=grpclog
time=“2025-01-16T15:13:45.617+00:00” level=trace msg=“[core] [Channel #280]Closing the name resolver” func=“config.(*gRPCLogger).Infoln” file=“config/logger.go:37” component=grpclog
time=“2025-01-16T15:13:45.617+00:00” level=trace msg=“[core] [Channel #280]ccBalancerWrapper: closing” func=“config.(*gRPCLogger).Infoln” file=“config/logger.go:37” component=grpclog
time=“2025-01-16T15:13:45.617+00:00” level=trace msg=“[core] [Channel #280 SubChannel #281]Subchannel Connectivity change to SHUTDOWN” func=“config.(*gRPCLogger).Infoln” file=“config/logger.go:37” component=grpclog
time=“2025-01-16T15:13:45.617+00:00” level=trace msg=“[core] [Channel #280 SubChannel #281]Subchannel deleted” func=“config.(*gRPCLogger).Infoln” file=“config/logger.go:37” component=grpclog
time=“2025-01-16T15:13:45.617+00:00” level=trace msg=“[transport] [client-transport 0xc00072a6c8] Closing: rpc error: code = Canceled desc = grpc: the client connection is closing” func=“config.(*gRPCLogger).Infoln” file=“config/logger.go:37” component=grpclog
time=“2025-01-16T15:13:45.617+00:00” level=trace msg=“[transport] [client-transport 0xc00072a6c8] loopyWriter exiting with error: rpc error: code = Canceled desc = grpc: the client connection is closing” func=“config.(*gRPCLogger).Infoln” file=“config/logger.go:37” component=grpclog
time=“2025-01-16T15:13:45.617+00:00” level=trace msg=“[core] [Channel #280]Channel deleted” func=“config.(*gRPCLogger).Infoln” file=“config/logger.go:37” component=grpclog
time=“2025-01-16T15:13:45.617+00:00” level=debug msg=“Connection closed.” func=client.dial.func1 file=“client/client.go:787” component=client
It doesn’t tell you what’s failing
Yes, it does:
time=“2025-01-16T15:13:45.617+00:00” level=debug msg=“Closing with error: rpc error: code = PermissionDenied desc = unexpected HTTP status code received from server: 403 (Forbidden);
Please try registering with the default admin user, and its new password.
time=“2025-01-16T15:24:37.768+00:00” level=info msg=“Connecting to https://admin:***@pmm.server.com.br:443/ …” func=client.dial file=“client/client.go:762” component=client
time=“2025-01-16T15:24:37.768+00:00” level=trace msg=“[core] original dial target is: "pmm.server.com.br:443"” func=“config.(*gRPCLogger).Infoln” file=“config/logger.go:37” component=grpclog
time=“2025-01-16T15:24:37.768+00:00” level=trace msg=“[core] [Channel #13]Channel created” func=“config.(*gRPCLogger).Infoln” file=“config/logger.go:37” component=grpclog
time=“2025-01-16T15:24:37.768+00:00” level=trace msg=“[core] [Channel #13]parsed dial target is: resolver.Target{URL:url.URL{Scheme:"passthrough", Opaque:"", User:(*url.Userinfo)(nil), Host:"", Path:"/pmm.server.com.br:443", RawPath:"", OmitHost:false, ForceQuery:false, RawQuery:"", Fragment:"", RawFragment:""}}” func=“config.(*gRPCLogger).Infoln” file=“config/logger.go:37” component=grpclog
time=“2025-01-16T15:24:37.768+00:00” level=trace msg=“[core] [Channel #13]Channel authority set to "pmm.server.com.br"” func=“config.(*gRPCLogger).Infoln” file=“config/logger.go:37” component=grpclog
time=“2025-01-16T15:24:37.768+00:00” level=trace msg=“[core] [Channel #13]Resolver state updated: {\n "Addresses": [\n {\n "Addr": "pmm.server.com.br:443",\n "ServerName": "",\n "Attributes": null,\n "BalancerAttributes": null,\n "Metadata": null\n }\n ],\n "Endpoints": [\n {\n "Addresses": [\n {\n "Addr": "pmm.server.com.br:443",\n "ServerName": "",\n "Attributes": null,\n "BalancerAttributes": null,\n "Metadata": null\n }\n ],\n "Attributes": null\n }\n ],\n "ServiceConfig": null,\n "Attributes": null\n} (resolver returned new addresses)” func=“config.(*gRPCLogger).Infoln” file=“config/logger.go:37” component=grpclog
time=“2025-01-16T15:24:37.768+00:00” level=trace msg=“[core] [Channel #13]Channel switches to new LB policy "pick_first"” func=“config.(*gRPCLogger).Infoln” file=“config/logger.go:37” component=grpclog
time=“2025-01-16T15:24:37.768+00:00” level=trace msg=“[pick-first-lb] [pick-first-lb 0xc0005ad5c0] Received new config {\n "shuffleAddressList": false\n}, resolver state {\n "Addresses": [\n {\n "Addr": "pmm.server.com.br:443",\n "ServerName": "",\n "Attributes": null,\n "BalancerAttributes": null,\n "Metadata": null\n }\n ],\n "Endpoints": [\n {\n "Addresses": [\n {\n "Addr": "pmm.server.com.br:443",\n "ServerName": "",\n "Attributes": null,\n "BalancerAttributes": null,\n "Metadata": null\n }\n ],\n "Attributes": null\n }\n ],\n "ServiceConfig": null,\n "Attributes": null\n}” func=“config.(*gRPCLogger).Infoln” file=“config/logger.go:37” component=grpclog
time=“2025-01-16T15:24:37.768+00:00” level=trace msg=“[core] [Channel #13 SubChannel #14]Subchannel created” func=“config.(*gRPCLogger).Infoln” file=“config/logger.go:37” component=grpclog
time=“2025-01-16T15:24:37.768+00:00” level=trace msg=“[core] [Channel #13]Channel Connectivity change to CONNECTING” func=“config.(*gRPCLogger).Infoln” file=“config/logger.go:37” component=grpclog
time=“2025-01-16T15:24:37.768+00:00” level=trace msg=“[core] [Channel #13]Channel exiting idle mode” func=“config.(*gRPCLogger).Infoln” file=“config/logger.go:37” component=grpclog
time=“2025-01-16T15:24:37.769+00:00” level=trace msg=“[core] [Channel #13 SubChannel #14]Subchannel Connectivity change to CONNECTING” func=“config.(*gRPCLogger).Infoln” file=“config/logger.go:37” component=grpclog
time=“2025-01-16T15:24:37.769+00:00” level=trace msg=“[core] [Channel #13 SubChannel #14]Subchannel picks a new address "pmm.server.com.br:443" to connect” func=“config.(*gRPCLogger).Infoln” file=“config/logger.go:37” component=grpclog
time=“2025-01-16T15:24:37.769+00:00” level=trace msg=“[pick-first-lb] [pick-first-lb 0xc0005ad5c0] Received SubConn state update: 0xc0005ad6b0, {ConnectivityState:CONNECTING ConnectionError: connectedAddress:{Addr: ServerName: Attributes: BalancerAttributes: Metadata:}}” func=“config.(*gRPCLogger).Infoln” file=“config/logger.go:37” component=grpclog
time=“2025-01-16T15:24:38.180+00:00” level=trace msg=“[core] [Channel #13 SubChannel #14]Subchannel Connectivity change to READY” func=“config.(*gRPCLogger).Infoln” file=“config/logger.go:37” component=grpclog
time=“2025-01-16T15:24:38.181+00:00” level=trace msg=“[pick-first-lb] [pick-first-lb 0xc0005ad5c0] Received SubConn state update: 0xc0005ad6b0, {ConnectivityState:READY ConnectionError: connectedAddress:{Addr:pmm.server.com.br:443 ServerName:pmm.server.com.br Attributes: BalancerAttributes: Metadata:}}” func=“config.(*gRPCLogger).Infoln” file=“config/logger.go:37” component=grpclog
time=“2025-01-16T15:24:38.181+00:00” level=trace msg=“[core] [Channel #13]Channel Connectivity change to READY” func=“config.(*gRPCLogger).Infoln” file=“config/logger.go:37” component=grpclog
time=“2025-01-16T15:24:38.181+00:00” level=info msg=“Connected to pmm.server.com.br:443.” func=client.dial file=“client/client.go:775” component=client
time=“2025-01-16T15:24:38.181+00:00” level=info msg=“Establishing two-way communication channel …” func=client.dial file=“client/client.go:796” component=client
time=“2025-01-16T15:24:38.181+00:00” level=debug msg=“Sending message (4 bytes): id:1 ping:{}.” func=“channel.(*Channel).send” file=“channel/channel.go:195” component=channel
time=“2025-01-16T15:24:38.321+00:00” level=debug msg=“Closing with error: rpc error: code = PermissionDenied desc = unexpected HTTP status code received from server: 403 (Forbidden); transport: received unexpected content-type "text/html"\nfailed to receive message\ngithub.com/percona/pmm/agent/client/channel.(*Channel).runReceiver\n\t/tmp/go/src/github.com/percona/pmm/agent/client/channel/channel.go:220\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1700” func=“channel.(*Channel).close.func1” file=“channel/channel.go:120” component=channel
time=“2025-01-16T15:24:38.321+00:00” level=debug msg=“Exiting receiver goroutine.” func=“channel.(*Channel).runReceiver.func1” file=“channel/channel.go:214” component=channel
time=“2025-01-16T15:24:38.321+00:00” level=error msg=“Failed to establish two-way communication channel: unexpected HTTP status code received from server: 403 (Forbidden); transport: received unexpected content-type "text/html".” func=client.dial file=“client/client.go:823” component=client
time=“2025-01-16T15:24:38.321+00:00” level=trace msg=“[core] [Channel #13]Channel Connectivity change to SHUTDOWN” func=“config.(*gRPCLogger).Infoln” file=“config/logger.go:37” component=grpclog
time=“2025-01-16T15:24:38.321+00:00” level=trace msg=“[core] [Channel #13]Closing the name resolver” func=“config.(*gRPCLogger).Infoln” file=“config/logger.go:37” component=grpclog
time=“2025-01-16T15:24:38.321+00:00” level=trace msg=“[core] [Channel #13]ccBalancerWrapper: closing” func=“config.(*gRPCLogger).Infoln” file=“config/logger.go:37” component=grpclog
time=“2025-01-16T15:24:38.322+00:00” level=trace msg=“[core] [Channel #13 SubChannel #14]Subchannel Connectivity change to SHUTDOWN” func=“config.(*gRPCLogger).Infoln” file=“config/logger.go:37” component=grpclog
time=“2025-01-16T15:24:38.322+00:00” level=trace msg=“[core] [Channel #13 SubChannel #14]Subchannel deleted” func=“config.(*gRPCLogger).Infoln” file=“config/logger.go:37” component=grpclog
time=“2025-01-16T15:24:38.322+00:00” level=trace msg=“[transport] [client-transport 0xc000a06008] Closing: rpc error: code = Canceled desc = grpc: the client connection is closing” func=“config.(*gRPCLogger).Infoln” file=“config/logger.go:37” component=grpclog
time=“2025-01-16T15:24:38.322+00:00” level=trace msg=“[transport] [client-transport 0xc000a06008] loopyWriter exiting with error: rpc error: code = Canceled desc = grpc: the client connection is closing” func=“config.(*gRPCLogger).Infoln” file=“config/logger.go:37” component=grpclog
time=“2025-01-16T15:24:38.322+00:00” level=trace msg=“[core] [Channel #13]Channel deleted” func=“config.(*gRPCLogger).Infoln” file=“config/logger.go:37” component=grpclog
time=“2025-01-16T15:24:38.322+00:00” level=debug msg=“Connection closed.” func=client.dial.func1 file=“client/client.go:787” component=client
I performed the test with the default user ‘admin’ and with the new password, it still gives the same error.
Is it possible that the error is on the PMM server?
It’s possible. Have you looked over any of the logs within the PMM server container? Check /srv/logs/