# Kubernetes ingress regression in PMM helm chart 1.2.0 **URL:** \ **Category:** PMM 2.x\ **Created:** [March 6, 2023, 1:14pm UTC](https://forums.percona.com/t/kubernetes-ingress-regression-in-pmm-helm-chart-1-2-0/20438 "2023-03-06T13:14:29Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![Artem\_Baguinski](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/artem_baguinski/32/7306_2.png) [@Artem\_Baguinski](https://forums.percona.com/u/Artem_Baguinski)\ **Post date:** [March 6, 2023, 1:14pm UTC](https://forums.percona.com/t/kubernetes-ingress-regression-in-pmm-helm-chart-1-2-0/20438/1 "2023-03-06T13:14:29Z")
PMM helm chart 1.2.0 introduces a breaking change in the ingress configuration. With default value of the `nginxInc` value (`false`) it assumes that something called “community managed ingress” is used which requires two separate ingresse configurations for pmm and pmm-grpc services. The implementation doesn’t work with e.g. traefik ingress controller (pmm paths work but pmm-grpc return 500 internal server error). Reverting to the helm chart 1.0.1 solves the problem (a singe ingress is used). I assume that setting `nginxInc` to `true` might have worked as well, but haven’t tested it because it would assume we are using nginx inc ingress controller, which we don’t. ---
**Author:** ![Michael\_Coburn](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/michael_coburn/32/18_2.png) [@Michael\_Coburn](https://forums.percona.com/u/Michael_Coburn)\ **Post date:** [March 7, 2023, 4:00pm UTC](https://forums.percona.com/t/kubernetes-ingress-regression-in-pmm-helm-chart-1-2-0/20438/2 "2023-03-07T16:00:30Z")
Hi @Artem_Baguinski , thank you for posting to the Percona forums! While I don’t have an answer for you, I have escalated your request in our Slack channel and you should get a reply here shortly 👍 ---
**Author:** ![Denys\_Kondratenko](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/denys_kondratenko/32/2798_2.png) [@Denys\_Kondratenko](https://forums.percona.com/u/Denys_Kondratenko)\ **Post date:** [March 7, 2023, 4:24pm UTC](https://forums.percona.com/t/kubernetes-ingress-regression-in-pmm-helm-chart-1-2-0/20438/3 "2023-03-07T16:24:23Z")
@Artem_Baguinski, thanks for letting us know. I had my concerns about that during my review. How does it affect you? It looks like just an additional endpoint that doesn’t work. ---
**Author:** ![Artem\_Baguinski](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/artem_baguinski/32/7306_2.png) [@Artem\_Baguinski](https://forums.percona.com/u/Artem_Baguinski)\ **Post date:** [March 8, 2023, 10:34am UTC](https://forums.percona.com/t/kubernetes-ingress-regression-in-pmm-helm-chart-1-2-0/20438/4 "2023-03-08T10:34:25Z")
Hi @Denys_Kondratenko Several endpoints don’t work, among them the one that ends with `/agent.`. As long as that is the case pmm-agent fails to register the services. The logs at that moment look like: ```auto Mar 06 11:16:58 db3 pmm-agent[73062]: DEBU[2023-03-06T11:16:58.394+01:00] Exiting receiver goroutine. component=channel Mar 06 11:16:58 db3 pmm-agent[73062]: ERRO[2023-03-06T11:16:58.394+01:00] Failed to establish two-way communication channel: unexpected HTTP status code received from server: 500 (Internal Server Error); transport: received unexpected content-type "text/plain; charset=utf-8". component=client Mar 06 11:16:58 db3 pmm-agent[73062]: DEBU[2023-03-06T11:16:58.394+01:00] Connection closed. component=client Mar 06 11:17:16 db3 pmm-agent[73062]: INFO[2023-03-06T11:17:16.246+01:00] Connecting to https://api_key:***@pmm.performation.cloud:443/ ... component=client Mar 06 11:17:16 db3 pmm-agent[73062]: INFO[2023-03-06T11:17:16.260+01:00] Connected to pmm.performation.cloud:443. component=client Mar 06 11:17:16 db3 pmm-agent[73062]: INFO[2023-03-06T11:17:16.260+01:00] Establishing two-way communication channel ... component=client Mar 06 11:17:16 db3 pmm-agent[73062]: DEBU[2023-03-06T11:17:16.260+01:00] Sending message (4 bytes): id:1 ping:{}. component=channel Mar 06 11:17:16 db3 pmm-agent[73062]: DEBU[2023-03-06T11:17:16.265+01:00] Closing with error: rpc error: code = Unknown desc = unexpected HTTP status code received from server: 500 (Internal Server Error); transport: received unexpected content-type "text/plain; charset=utf-8" Mar 06 11:17:16 db3 pmm-agent[73062]: failed to receive message Mar 06 11:17:16 db3 pmm-agent[73062]: github.com/percona/pmm/agent/client/channel.(*Channel).runReceiver Mar 06 11:17:16 db3 pmm-agent[73062]: /tmp/go/src/github.com/percona/pmm/agent/client/channel/channel.go:220 Mar 06 11:17:16 db3 pmm-agent[73062]: runtime.goexit ``` ---
**Author:** ![Denys\_Kondratenko](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/denys_kondratenko/32/2798_2.png) [@Denys\_Kondratenko](https://forums.percona.com/u/Denys_Kondratenko)\ **Post date:** [March 27, 2023, 11:43am UTC](https://forums.percona.com/t/kubernetes-ingress-regression-in-pmm-helm-chart-1-2-0/20438/5 "2023-03-27T11:43:44Z")
[https://jira.percona.com/browse/PMM-11872](https://jira.percona.com/browse/PMM-11872) working on a fix ---
**Author:** ![jeff.plewes](https://avatars.discourse-cdn.com/v4/letter/j/7bcc69/32.png) [@jeff.plewes](https://forums.percona.com/u/jeff.plewes)\ **Post date:** [September 29, 2023, 5:10pm UTC](https://forums.percona.com/t/kubernetes-ingress-regression-in-pmm-helm-chart-1-2-0/20438/6 "2023-09-29T17:10:14Z")
A work-around i provided to [PMM Server deployed on k8s - failed to establish two-way communication channel - #5 by jeff.plewes](https://forums.percona.com/t/pmm-server-deployed-on-k8s-failed-to-establish-two-way-communication-channel/22708/5) Add this to your helm values for ingress: ```auto community: annotations: nginx.ingress.kubernetes.io/use-regex: "true" ``` The [nginx.ingress.kubernetes.io/use-regex:](http://nginx.ingress.kubernetes.io/use-regex:) “true” was needed to have the ingress controller use regex location blocks for ‘/agent., /inventory.’ etc… without this config, such calls to these urls would not match the location blocks in the ingress controller and instead hit the default location / and its non-grpc upstream. ---
**Author:** ![Artem\_Baguinski](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/artem_baguinski/32/7306_2.png) [@Artem\_Baguinski](https://forums.percona.com/u/Artem_Baguinski)\ **Post date:** [October 12, 2023, 4:59am UTC](https://forums.percona.com/t/kubernetes-ingress-regression-in-pmm-helm-chart-1-2-0/20438/7 "2023-10-12T04:59:08Z")
Hi Jeff Thanks for the suggestion. However it wouldn’t work for us, as we don’t use an ingress controller based on nginx. Cheers, Artem. ---
**Author:** ![Miguel\_Zenteno](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/miguel_zenteno/32/21039_2.png) [@Miguel\_Zenteno](https://forums.percona.com/u/Miguel_Zenteno)\ **Post date:** [May 19, 2025, 10:33am UTC](https://forums.percona.com/t/kubernetes-ingress-regression-in-pmm-helm-chart-1-2-0/20438/8 "2025-05-19T10:33:10Z")
@Aaditya_Dubey is NGINX the only way to make this work? I’m was trying to achieve this with AWS ALBs, but the GRPC nignx annotations are hardcoded in the helm chart (at least until 1.3.21).