I am running Percona PMM 3.4.1 deployed via Docker. I encountered an issue where vmalert cannot start at all. The error message indicates that neither -notifier.url, -notifier.config, nor -notifier.blackhole is configured.
The problem is that the supervisor configuration for vmalert is managed by pmm-managed, and any manual changes I make to the config files are automatically reverted.
My goal is to stop using PMM for alerts and instead let Nightingale (N9e) handle alerting, while keeping the VictoriaMetrics port 9090 exposed so that N9e can read monitoring data directly.
I want to either:
Properly start vmalert with a notifier, or
Understand if there is a supported way to disable it safely without affecting PMM functionality, so that N9e can take over alerting while still accessing the 9090 Prometheus-compatible endpoint.
It seems that our situation is as follows: we were originally using PMM 2 and recently upgraded to PMM 3.4.1 by redeploying a new container. We also copied over the alert rule files we were using previously into the prometheus/rules directory of the new PMM 3.4.1 deployment.
In this setup, when starting vmalert, it immediately fails with an error indicating that -notifier.url, -notifier.config, or -notifier.blackhole is not configured. Additionally, the vmalert supervisor configuration is managed by pmm-managed, so any manual changes are automatically reverted.
2025-11-10T04:21:54.557Z info /home/builder/rpm/BUILD/VictoriaMetrics-pmm-6401-v1.114.0/lib/logger/flag.go:20 -datasource.url="secret"
2025-11-10T04:21:54.557Z info /home/builder/rpm/BUILD/VictoriaMetrics-pmm-6401-v1.114.0/lib/logger/flag.go:20 -external.url="http://127.0.0.1:9090/prometheus/"
2025-11-10T04:21:54.557Z info /home/builder/rpm/BUILD/VictoriaMetrics-pmm-6401-v1.114.0/lib/logger/flag.go:20 -httpListenAddr="127.0.0.1:8880"
2025-11-10T04:21:54.557Z info /home/builder/rpm/BUILD/VictoriaMetrics-pmm-6401-v1.114.0/lib/logger/flag.go:20 -remoteRead.url="secret"
2025-11-10T04:21:54.557Z info /home/builder/rpm/BUILD/VictoriaMetrics-pmm-6401-v1.114.0/lib/logger/flag.go:20 -remoteWrite.url="secret"
2025-11-10T04:21:54.557Z info /home/builder/rpm/BUILD/VictoriaMetrics-pmm-6401-v1.114.0/lib/logger/flag.go:20 -rule="/srv/prometheus/rules/*.yml"
2025-11-10T04:21:54.559Z info /home/builder/rpm/BUILD/VictoriaMetrics-pmm-6401-v1.114.0/app/vmalert/main.go:163 reading rules configuration file from "/srv/prometheus/rules/*.yml"
2025-11-10T04:21:54.559Z info /home/builder/rpm/BUILD/VictoriaMetrics-pmm-6401-v1.114.0/app/vmalert/config/log/logger.go:52 found 19 files to read from "Local FS{MatchPattern: \"/srv/prometheus/rules/*.yml\"}"
2025-11-10T04:21:54.560Z info /home/builder/rpm/BUILD/VictoriaMetrics-pmm-6401-v1.114.0/app/vmalert/config/log/logger.go:52 finished reading 19 files in 537.383µs from "Local FS{MatchPattern: \"/srv/prometheus/rules/*.yml\"}"
2025-11-10T04:21:54.672Z fatal /home/builder/rpm/BUILD/VictoriaMetrics-pmm-6401-v1.114.0/app/vmalert/main.go:175 failed to start: config contains alerting rules but neither `-notifier.url` nor `-notifier.config` nor `-notifier.blackhole` aren't set
When we deleted the contents of the prometheus/rules directory, vmalert was able to start. So, how exactly is vmalert intended to be used in PMM 3? It shouldn’t be causing this kind of problem. Additionally, the PMM UI no longer has the graph/settings/am-integration page where this could be configured.
I looked at the code briefly and think I understand now. Basically, if vmalert fails to start, pmm-managed will keep looping through the setup process and regenerate the configuration file.
if !setup(ctx, deps) {
go func() {
const delay = 2 * time.Second
for {
deps.l.Warnf("Retrying in %s.", delay)
sleepCtx, sleepCancel := context.WithTimeout(ctx, delay)
<-sleepCtx.Done()
sleepCancel()
if ctx.Err() != nil {
return
}
if setup(ctx, deps) {
return
}
}
}()
}
So, if vmalert starts normally, I can manually modify the supervisor config file, and unless I restart the container, the setup process will not be triggered again.