Transport: authentication handshake failed: remote error: tls: no application protocol

Percona Monitoring and Management (PMM) PMM 2.x
1

I have searched the web and not found an answer to this…

Basically I am going though a Citrix Loadbalancer (I have to use them)

I can connect to the server directly and items run properly… When I try the https configure command I get

Jul 19 16:01:33 servername pmm-agent: #033[33mWARN#033[0m[2023-07-19T16:01:33.464-04:00]grpclog/grpclog.go:46 grpclog.WarningDepth [core] [Channel #529 SubChannel #530] grpc: addrConn.createTransport failed to connect to {Addr: "pmm-server.com:443", ServerName: "pmm-server.com", }. Err: connection error: desc = "transport: authentication handshake failed: remote error: tls: no application protocol"  #033[33mcomponent#033[0m=grpclog
Jul 19 16:01:33 servername pmm-agent: #033[37mTRAC#033[0m[2023-07-19T16:01:33.464-04:00]config/logger.go:37 config.(*gRPCLogger).Infoln [core] [Channel #529 SubChannel #530] Subchannel Connectivity change to TRANSIENT_FAILURE, last error: connection error: desc = "transport: authentication handshake failed: remote error: tls: no application protocol"  #033[37mcomponent#033[0m=grpclog
Jul 19 16:01:33 servername pmm-agent: #033[37mTRAC#033[0m[2023-07-19T16:01:33.464-04:00]config/logger.go:37 config.(*gRPCLogger).Infoln [core] pickfirstBalancer: UpdateSubConnState: 0xc0002faf60, {TRANSIENT_FAILURE connection error: desc = "transport: authentication handshake failed: remote error: tls: no application protocol"}  #033[37mcomponent#033[0m=grpclog
Jul 19 16:01:35 servername pmm-agent: #033[37mTRAC#033[0m[2023-07-19T16:01:35.007-04:00]config/logger.go:37 config.(*gRPCLogger).Infoln [core] [Channel #529 SubChannel #530] Subchannel Connectivity change to IDLE, last error: connection error: desc = "transport: authentication handshake failed: remote error: tls: no application protocol"  #033[37mcomponent#033[0m=grpclog
Jul 19 16:01:35 servername pmm-agent: #033[37mTRAC#033[0m[2023-07-19T16:01:35.007-04:00]config/logger.go:37 config.(*gRPCLogger).Infoln [core] pickfirstBalancer: UpdateSubConnState: 0xc0002faf60, {IDLE connection error: desc = "transport: authentication handshake failed: remote error: tls: no application protocol"}  #033[37mcomponent#033[0m=grpclog
Jul 19 16:01:35 servername pmm-agent: #033[37mTRAC#033[0m[2023-07-19T16:01:35.007-04:00]config/logger.go:37 config.(*gRPCLogger).Infoln [core] [Channel #529 SubChannel #530] Subchannel Connectivity change to CONNECTING  #033[37mcomponent#033[0m=grpclog
Jul 19 16:01:35 servername pmm-agent: #033[37mTRAC#033[0m[2023-07-19T16:01:35.007-04:00]config/logger.go:37 config.(*gRPCLogger).Infoln [core] [Channel #529 SubChannel #530] Subchannel picks a new address "pmm-server.com:443" to connect  #033[37mcomponent#033[0m=grpclog
Jul 19 16:01:35 servername pmm-agent: #033[37mTRAC#033[0m[2023-07-19T16:01:35.007-04:00]config/logger.go:37 config.(*gRPCLogger).Infoln [core] pickfirstBalancer: UpdateSubConnState: 0xc0002faf60, {CONNECTING <nil>}  #033[37mcomponent#033[0m=grpclog
Jul 19 16:01:35 servername pmm-agent: #033[37mTRAC#033[0m[2023-07-19T16:01:35.074-04:00]config/logger.go:37 config.(*gRPCLogger).Infoln [core] Creating new client transport to "{Addr: \"pmm-server.com:443\", ServerName: \"pmm-server.com\", }": connection error: desc = "transport: authentication handshake failed: remote error: tls: no application protocol"  #033[37mcomponent#033[0m=grpclog

I use this command

pmm-admin config --server-insecure-tls --server-url=https://xxxx:xxxxx@pmm-server.comm:443 --force --log-level=error --region=“xx” --custom-labels=“location=xx” --trace --debug

but pmm-admin status shows a connected of false (when using the Netscaler) using the server direct everything works.

I have not found a hit anyplace on items similar to this… “transport: authentication handshake failed: remote error: tls: no application protocol”

Any suggestions?

**Note the Netscaler has a solid cert via a internet company (not self signed)

2

You will need to configure your netscaler to forward “gRPC over HTTPS” requests for several contexts. If you look inside your pmm container (or AMI/OVF) in /etc/nginx/conf.d/pmm.conf you’ll see the grpc contexts that you need to handle (search grpc_pass in that file). This might help: gRPC end-to-end configuration | NetScaler 13.1

forgot to add that the UI itself is just plain HTTP(s) so it should probably work fine accessing the UI via the netscaler but client registration and metrics streaming both use gRPC and needs extra config for external load balancers to pass that traffic correctly.

1 Like
3

Wow… ok I thought I looked pretty close for some clue on this…

So if I am following this correctly the netscaler needs to account for these 4 items? I did not know much about grpc until well… today…

    # pmm-managed gRPC APIs
    location /agent. {
      grpc_pass grpc://managed-grpc;
      # Disable request body size check for gRPC streaming, see https://trac.nginx.org/nginx/ticket/1642.
      # pmm-managed uses grpc.MaxRecvMsgSize for that.
      client_max_body_size 0;
    }
    location /inventory. {
      grpc_pass grpc://managed-grpc;
    }
    location /management. {
      grpc_pass grpc://managed-grpc;
    }
    location /server. {
      grpc_pass grpc://managed-grpc;
    }
4

Yes, that should do it having those four forwarded to use the netscaler as a reverse proxy/load balancer in front of PMM. There may be some SSL headaches you need to handle as well because you’re terminating one SSL connection at the netscaler and creating a brand new one from netscaler to PMM’s nginx. That usually involves pre-installing/trusting the certs that PMM presents (which are self-signed by default so security but far from the best security). I am pretty sure there are past forum posts on this as well.

1 Like