Unable to connect ipv6 host to dual stacked pmm host

Percona Monitoring and Management (PMM)
1

Description:

vmagent tries to connect via ipv4 even if the own host does not support it.

Steps to Reproduce:

We have two hosts

  1. MariaDB node running pmm-agent. This host only supports ipv6.
  2. PMM node. This host supports ipv4 and ipv6.

We are not able to see node/mysql metrics in PMM or VictoriaMetrics. Running pmm-agent in debug mode with debug logs shows

May 21 13:56:11 db1 pmm-agent[1265516]: time="2024-05-21T13:56:11.343+02:00" level=info msg="2024-05-21T11:56:11.342Z\twarn\tVictoriaMetrics/app/vmagent/remotewrite/client.go:385\tcouldn't send a block with size 31797 bytes to \"1:secret-url\": Post \"https://pmm-host:443/victoriametrics/api/v1/write\": dial tcp4 1.2.3.4:443: connect: network is unreachable; re-sending the block in 60.000 seconds" agentID=/agent_id/a-b-c-d-e-f component=agent-process type=vm_agent

1.2.3.4 is the redacted IPv4 address of the pmm host.

Version:

pmm-agent 2.41.2

Logs:

May 21 13:56:11 db1 pmm-agent[1265516]: time="2024-05-21T13:56:11.343+02:00" level=info msg="2024-05-21T11:56:11.342Z\twarn\tVictoriaMetrics/app/vmagent/remotewrite/client.go:385\tcouldn't send a block with size 31797 bytes to \"1:secret-url\": Post \"https://pmm-host:443/victoriametrics/api/v1/write\": dial tcp4 1.2.3.4:443: connect: network is unreachable; re-sending the block in 60.000 seconds" agentID=/agent_id/e2d2bc6d-4ca5-cb4d-85d6-be46323eee04 component=agent-process type=vm_agent

Expected Result:

vmagent can connect to the pmm host via ipv6.

Actual Result:

vmagent tries ipv4 and fails.

Additional Information:

What we have tried already

  1. Adding an /etc/hosts file on the db host with only the ipv6 address. This should avoid a DNS lookup which returns the IPv4 address too.
  2. Setting VMAGENT_enableTCP6=true for the pmm-agent but it does not seem to forward it to vmagent despite PMM-9374 external victoria metrics (#1916) · percona/pmm@ffc66f0 · GitHub.
2

Reported at VMAGENT_ env vars not forwarded to vmagent (to enable TCP6) · Issue #3013 · percona/pmm · GitHub.

3

The environment variable has to be set on the PMM server, not the client. The server will distribute the configuration to all clients.