[PMM 3.1.0] Alertmanager TLS Certificate Trust Issue with Proxy (Slack Webhook)

[pmm3.1]

I’m currently using PMM 3.1.0 (Docker) and trying to configure Alertmanager to send Slack notifications through a corporate proxy server. However, I’m facing a TLS issue due to a self-signed (or internally signed) certificate on the proxy.

Goal :
To allow PMM Alertmanager to successfully send Slack webhook notifications through a corporate HTTPS proxy (e.g., proxy.example.com:3333) which uses a custom certificate authority (CA).

What I’ve tried so far

1. insecure_skip_verify: true in alertmanager.base.yml

receivers: 
- name: 'slack' 
slack_configs: 
- api_url: 'https://hooks.slack.com/services/XXX/YYYY/ZZZ' 
channel: '#alerts' 
send_resolved: true 
http_config: 
tls_config: 
insecure_skip_verify: true

→ Restarted pmm-managed with supervisorctl restart pmm-managed
→ No effect. TLS x509 error persists.

2. Specifying CA certificate file via ca_file
   Copied internally trusted proxy-ca-cert.crt to /srv/alertmanager/

Updated config:

http_config: 
tls_config: 
ca_file: /srv/alertmanager/proxy-ca-cert.crt

→ Still not applied. Alertmanager does not appear to respect this setting.

3. System-wide CA trust update attempt
   Placed .crt into /etc/pki/ca-trust/source/anchors/

Ran update-ca-trust extract

→ Fails with:
p11-kit: couldn’t create file … Permission denied
(possibly due to container’s limited root access)

Questions :
In PMM 3.1.0, how can we properly configure Alertmanager to:

1. skip TLS verification (like, --insecure),
2. trust a custom certificate (e.g., for proxy use)?
3. Does alertmanager.base.yml file is working in PMM 3.x alertmanager?

Environment
PMM version: 3.1.0
Deployment: Docker (manually run container)
Outbound proxy: proxy.example.com:3333 (with internal CA)

Error message:
x509: certificate signed by unknown authority

How can I handle Slack messaging through my company proxy server?
Any guidance on how to proceed would be greatly appreciated.
Thanks in advance for your help!

[nurlan]

Hello, we dropped support of internal alertmanager in PMM 3, currently we support external alertmanagers only through Grafana UI.

x509: certificate signed by unknown authority

Can you share where do you see this error?

[pmm3.1]

Hello, This proxy uses an internally signed certificate, which results in the following TLS error when sending test alerts

What I’ve learned so far:

• I understand that PMM 3.x has deprecated internal Alertmanager, and that all alerting is now handled via Grafana Alerting (Unified Alerting).
• The error above seems to originate from Grafana’s Slack notification contact point, which attempts to send a message through the proxy but fails to validate the proxy’s TLS certificate.

What would be the recommended way to allow Slack notifications to pass through the proxy without TLS failures?

While investigating further, I noticed that vmalert is running as part of the PMM 3.1.0 container, alongside Grafana and other components.

Does vmalert handle any of the alert notification delivery (e.g., Slack)?

• Or is all outbound alerting handled exclusively by Grafana Alerting / Contact Points in PMM 3.x?

Additionally, if vmalert is involved in sending alerts externally:

• Is it possible to configure vmalert to use a corporate proxy for outbound requests (e.g., via environment variables like HTTP_PROXY, or via CLI flags)?
• Can vmalert be configured to trust a custom CA certificate for TLS communication (similar to how system CA bundles are used)?

Thanks again for your continued support!

[nurlan]

Hi,
we left vmalert only for recording rules. We had plans to use recording rules for better performance. it doesn’t communicate with Grafana unified alerting.

I’ve found Github issue which might be related to your one