After upgrading from PMM 2.18 to PMM 2.19 all pmm-agent seems to forget the configured port for external services (note the “…127.0.0.1:0/metrics” - port should be 42009).
Example log entry:
Jul 13 10:48:18 envi-test-rmq-1 pmm-agent[1465384]: INFO[2021-07-13T10:48:18.935+02:00] 2021-07-13T08:48:18.935Z error VictoriaMetrics/lib/promscrape/scrapework.go:231 error when scraping “http://127.0.0.1:0/metrics” from job “external-exporter_agent_id_a856cbdb-4a4b-442b-b9af-62957ca2477b_mr-10s” with labels {agent_id=“/agent_id/a856cbdb-4a4b-442b-b9af-62957ca2477b”,agent_type=“external-exporter”,external_group=“external”,instance=“/agent_id/a856cbdb-4a4b-442b-b9af-62957ca2477b”,job=“external-exporter_agent_id_a856cbdb-4a4b-442b-b9af-62957ca2477b_mr-10s”,machine_id=“/machine_id/a69c5724b306422387bb1b08ee1f2edb”,node_id=“/node_id/f5dd0e77-405a-4bcf-a3c5-7e82652f954e”,node_name=“envi-test-rmq-1”,node_type=“generic”,service_id=“/service_id/ec7a45a9-025f-4075-876c-d116c0b6f394”,service_name=“envi-test-rmq-1-systemd”,service_type=“external”}: error when scraping “http://127.0.0.1:0/metrics”: dial tcp4 127.0.0.1:0: connect: connection refused; try -enableTCP6 command-line flag if you scrape ipv6 addresses agentID=/agent_id/5c7d46da-b037-4cb9-8e75-02e37de04f1c component=agent-process type=vm_agent
After removing the service and adding it again everything seems to work properly - until pmm-agent is being restarted. Then the same error occurs.
In this example the external service is added with
pmm-admin add external --metrics-path=metrics --service-name=“envi-test-rmq-1-systemd” --listen-port=42009 --scheme=http --metrics-mode=auto
This issue occurs with different pmm-client versions - at least between 2.15 and 2.19.