Grpc: the client connection is closing

Percona Monitoring and Management (PMM)
1

Description:

Deploying pmm on EKS cluster via helm chart.

for Ingress I’m using ALB with the following configuration with the AWS ingress controller

    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/subnets: "__PUBLIC_SUBNET_1__, __PUBLIC_SUBNET_2__, __PUBLIC_SUBNET_3__"
    alb.ingress.kubernetes.io/security-groups: "__EKS_SG__"
    alb.ingress.kubernetes.io/backend-protocol: HTTP
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443,"HTTP":80}]'
    alb.ingress.kubernetes.io/healthcheck-path: /healthz
    alb.ingress.kubernetes.io/success-codes: 200-499
    alb.ingress.kubernetes.io/certificate-arn: __CERTIFICATE_ARN__
    alb.ingress.kubernetes.io/ssl-policy: ELBSecurityPolicy-TLS13-1-2-Res-2021-06
    alb.ingress.kubernetes.io/actions.redirect: >-
      {"Type":"redirect","RedirectConfig":{"Protocol":"HTTPS","Port":"443","StatusCode":"HTTP_301"}}
    alb.ingress.kubernetes.io/load-balancer-attributes: >-
      idle_timeout.timeout_seconds=120,
      routing.http.drop_invalid_header_fields.enabled=true,
      deletion_protection.enabled=true

I get these two errors in the status of pmm-agent

time="2026-06-08T07:58:23.128+00:00" level=error msg="Failed to create gRPC stream to Agents service: failed to receive message: rpc error: code = Unknown desc = unexpected HTTP status code received from server: 464 (); malformed header: missing HTTP content-type." component=client

time="2026-06-08T07:58:23.128+00:00" level=error msg="Connection closed: rpc error: code = Canceled desc = grpc: the client connection is closing." component=client

The agent is being registered successfully but the node is shown as “Unknown” so I guess e can rule out network configuration.

I’ve also added second ingress for grpc wiht this config

kind: Ingress
metadata:
  name: rg-percona-monitoring-grpc
  namespace: rg-percona-monitoring
  labels:
    app: rg-percona-monitoring
    env: testing
    srv: grpc
  annotations:
    # Share the same ALB using group name
    alb.ingress.kubernetes.io/group.name: rg-percona-monitoring
    alb.ingress.kubernetes.io/group.order: '5'

    # Use existing ALB settings
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/certificate-arn: __CERTIFICATE_ARN__
    alb.ingress.kubernetes.io/ssl-policy: ELBSecurityPolicy-TLS13-1-2-Res-2021-06
    alb.ingress.kubernetes.io/security-groups: __EKS_SG__
    alb.ingress.kubernetes.io/subnets: __PUBLIC_SUBNET_1__, __PUBLIC_SUBNET_2__, __PUBLIC_SUBNET_3__

    # gRPC specific configuration
    alb.ingress.kubernetes.io/backend-protocol: HTTPS
    alb.ingress.kubernetes.io/backend-protocol-version: GRPC
    alb.ingress.kubernetes.io/target-type: ip

    # gRPC health check
    alb.ingress.kubernetes.io/healthcheck-protocol: HTTPS
    alb.ingress.kubernetes.io/healthcheck-path: /ping
    alb.ingress.kubernetes.io/healthcheck-interval-seconds: '30'
    alb.ingress.kubernetes.io/healthcheck-timeout-seconds: '10'
    alb.ingress.kubernetes.io/healthy-threshold-count: '2'
    alb.ingress.kubernetes.io/unhealthy-threshold-count: '3'
    alb.ingress.kubernetes.io/success-codes: '0,12'

    # Only HTTPS for gRPC
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}]'

    # Increased timeout for persistent gRPC connections
    alb.ingress.kubernetes.io/load-balancer-attributes: idle_timeout.timeout_seconds=120

    # Condition: Only route gRPC traffic (Content-Type header)
    alb.ingress.kubernetes.io/conditions.grpc: |
      [{"field":"http-header","httpHeaderConfig":{"httpHeaderName":"Content-Type","values":["application/grpc","application/grpc+proto"]}}]

Steps to Reproduce:

systemctl start pmm-agent

or

/usr/local/percona/pmm2/bin/pmm-agent run --config-file=/usr/local/percona/pmm/config/pmm-agent.yaml

Version:

PMM 3.8.0