Pmm-client not connecting to pmm-server with TLS (EKS/k8s setup)

Percona Monitoring and Management (PMM) PMM 2.x
1

Description:

Hey team, I have pmm server running in aws EKS with ingress class ALB and this is my helm values for the service and ingress setup with tls.

  service:
    type: NodePort
    name: pmm-service
    ports:
      - port: 443
        targetPort: 443
        protocol: TCP
        name: https
      - port: 80
        targetPort: 80
        protocol: TCP
        name: http

  ingress:
    enabled: true
    ingressClassName: "alb"
    nginxInc: true
    pathType: Prefix
    annotations:
      alb.ingress.kubernetes.io/scheme: internal
      alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}]'
      alb.ingress.kubernetes.io/ssl-redirect: "443"
      alb.ingress.kubernetes.io/target-type: ip
      alb.ingress.kubernetes.io/certificate-arn: ****
      alb.ingress.kubernetes.io/healthcheck-path: /v1/readyz
      alb.ingress.kubernetes.io/healthcheck-protocol: HTTP
      alb.ingress.kubernetes.io/backend-protocol: HTTP
      alb.ingress.kubernetes.io/security-groups: ****
      alb.ingress.kubernetes.io/load-balancer-attributes: idle_timeout.timeout_seconds=3600
      alb.ingress.kubernetes.io/grpc-pass-through: "true"
      alb.ingress.kubernetes.io/conditions.grpc-content-type: '["application/grpc+proto","application/grpc"]'
    hosts:
      - host: pmm.tooling.net
        paths:
          - /
    tls:
      - hosts:
          - pmm.tooling.net
        secretName: pmm-tls

I have pmm.tooling.net DNS as route53 alias record created for the load balancer endpoint and I can access the admin UI with https.
my setup is pmm-server(EKS) → Ingress ALB → route53-alias URL → pmm-clients(EC2) /admin UI
but when I update the pmm-client running on ec2 node to send data to pmm.tooling.net i see the error that I have posted under logs.
I have verified the network and can also ssl verify the dns from the ec2 mysql node.

Version:

2.42

Logs:

pmm-agent[15865]: ERRO[2024-08-30T04:38:43.610-07:00] Failed to establish two-way communication channel: unexpected HTTP status code received from server: 464 (); malformed header: missing HTTP content-type. component=client

Additional Information:

Had also tried with nginxInc: false which then creates a separate grpc ingress with the main one and I am not sure which one are we supposed to use for the clients.
The backend port from the pmm helm is set to http only the grpc ports are supposed to use https only

this is deployed ingress spec of manifest with a single ingress with alb type with nginxInc: true

spec:
  ingressClassName: alb
  rules:
    - host: pmm.tooling.net
      http:
        paths:
          - backend:
              service:
                name: pmm-service
                port:
                  number: 80
            path: /
            pathType: Prefix
          - backend:
              service:
                name: pmm-service-grpc
                port:
                  number: 443
            path: /agent.
            pathType: Prefix
          - backend:
              service:
                name: pmm-service-grpc
                port:
                  number: 443
            path: /inventory.
            pathType: Prefix
          - backend:
              service:
                name: pmm-service-grpc
                port:
                  number: 443
            path: /management.
            pathType: Prefix
          - backend:
              service:
                name: pmm-service-grpc
                port:
                  number: 443
            path: /server.
            pathType: Prefix
  tls:
    - hosts:
        - pmm.tooling.net
      secretName: pmm-tls

can someone suggest the right approach to get clients send data to pmm-server with tls, Thank you.