I’ve updated to the latest 2.25 version, and I’m seeing these messages scroll over and over in /srv/logs/pmm-managed.log
WARN[2022-01-14T15:55:48.151+00:00] Setting global email config, any user defined changes to the base config might be overwritten. component=alertmanager
ERRO[2022-01-14T15:55:48.182+00:00] Checking '/tmp/pmm-managed-config-alertmanager-076327072' FAILED: address 127.0.0.1: missing port in address
amtool: error: failed to validate 1 file(s)
component=alertmanager
ERRO[2022-01-14T15:55:48.182+00:00] Failed to update configuration, will retry: exit status 1
Checking '/tmp/pmm-managed-config-alertmanager-076327072' FAILED: address 127.0.0.1: missing port in address
amtool: error: failed to validate 1 file(s)
github.com/percona/pmm-managed/services/alertmanager.(*Service).validateConfig
/home/builder/rpm/BUILD/pmm-managed-dc091abb30f1e0ac8769a02d0587aa1be653a58c/src/github.com/percona/pmm-managed/services/alertmanager/alertmanager.go:292
github.com/percona/pmm-managed/services/alertmanager.(*Service).configAndReload
/home/builder/rpm/BUILD/pmm-managed-dc091abb30f1e0ac8769a02d0587aa1be653a58c/src/github.com/percona/pmm-managed/services/alertmanager/alertmanager.go:325
github.com/percona/pmm-managed/services/alertmanager.(*Service).updateConfiguration
/home/builder/rpm/BUILD/pmm-managed-dc091abb30f1e0ac8769a02d0587aa1be653a58c/src/github.com/percona/pmm-managed/services/alertmanager/alertmanager.go:205
github.com/percona/pmm-managed/services/alertmanager.(*Service).Run
/home/builder/rpm/BUILD/pmm-managed-dc091abb30f1e0ac8769a02d0587aa1be653a58c/src/github.com/percona/pmm-managed/services/alertmanager/alertmanager.go:173
main.main.func7
/home/builder/rpm/BUILD/pmm-managed-dc091abb30f1e0ac8769a02d0587aa1be653a58c/src/github.com/percona/pmm-managed/main.go:841
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1371. component=alertmanager
I do not have “Integrated Alerting” enabled, and I’m not sure where this is coming from. Any suggestions?
Hi Danny,
Could check any errors in next log files?
/srv/logs/alertmanager.log
/srv/logs/pmm-update-perform.log
Hey Vadim.
Here is: /srv/logs/alertmanager.log
level=info ts=2022-01-14T15:47:44.638Z caller=main.go:524 msg="Received SIGTERM, exiting gracefully..."
level=info ts=2022-01-14T15:47:48.627Z caller=main.go:216 msg="Starting Alertmanager" version="(version=0.21.0, branch=non-git, revision=non-git)"
level=info ts=2022-01-14T15:47:48.627Z caller=main.go:217 build_context="(go=go1.15.7, user=builder@b33c58619690, date=20210517-08:24:43)"
level=info ts=2022-01-14T15:47:48.671Z caller=coordinator.go:119 component=configuration msg="Loading configuration file" file=/etc/alertmanager.yml
level=info ts=2022-01-14T15:47:48.671Z caller=coordinator.go:131 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager.yml
level=info ts=2022-01-14T15:47:48.674Z caller=main.go:485 msg=Listening address=127.0.0.1:9093
Contents of /etc/alertmanager.yml
---
# You can edit this file; changes will be preserved.
route:
receiver: empty
routes: []
receivers:
- name: empty
No errors in /srv/logs/pmm-update-perform.log that I can see.
I’m assuming Alertmanager and Integrated Alerting are two separate things, so with Integrated Alerting shut off it makes no difference. However, I didn’t configure anything specific for Alertmanager, so not sure why that would be having issues.
Hi @dbray_sd
Did you modify the below file, can you please check once in the below file?
cat /etc/grafana/grafana.ini
May be this might be the reason, I am not sure but just try to check this once.
It looks like the configuration files is protected for writing.
So pmm-managed service can’t recreate it.
e.g.
Could you remove this file and restart pmm-managed service?
Hey @Naresh9999
No, I did not manually modify that file.
[smtp]
;enabled = false
;host = localhost:25
;user =
# If the password contains # or ; you have to wrap it with triple quotes. Ex """#password;"""
;password =
;cert_file =
;key_file =
;skip_verify = false
;from_address = admin@grafana.localhost
;from_name = Grafana
# EHLO identity in SMTP dialog (defaults to instance_name)
;ehlo_identity = dashboard.example.com
# SMTP startTLS policy (defaults to 'OpportunisticStartTLS')
;startTLS_policy = NoStartTLS
@Vadim_Yalovets
Odd, it recreated it…but recreated it with the exact same contents:
[root@1fd26d7c5d00 opt]# supervisorctl stop pmm-managed
pmm-managed: stopped
[root@1fd26d7c5d00 opt]# rm /etc/alertmanager.yml
rm: remove regular file ‘/etc/alertmanager.yml’? y
[root@1fd26d7c5d00 opt]# supervisorctl start pmm-managed
pmm-managed: started
[root@1fd26d7c5d00 opt]# cat /etc/alertmanager.yml
---
# You can edit this file; changes will be preserved.
route:
receiver: empty
routes: []
receivers:
- name: empty
Hi @dbray_sd
Can you stop and start the docker?
@Naresh9999
Already tried that. It still recreates the file with the same “empty” contents as above.
Thanks.
Hi @dbray_sd
Is the below file still exists with the same name?
/tmp/pmm-managed-config-alertmanager-076327072
can you remove it and then restart the pmm server?
My file was named:
/tmp/pmm-managed-config-alertmanager-087463742
…I removed it and /etc/alertmanager.yml restarted. Problem remains, /etc/alertmanager.yml recreated with same empty contents, and no new /tmp/pmm-managed-config-alertmanager-* file was created.
Did you enable the Alerting in the GUI before upgrade to 2.25?
If you mean the “Integrated Alerting”, yes, that was enabled, but was never used/configured. So it was disabled after upgrading to 2.25. However, this issue was happening before the upgrade to 2.25.
Can you disable(in gui) it now and then restart the pmm server?
Already tried that as well. It has been disabled this entire time in the gui, before I posted to this forum. I’ve also enabled it restarted, disabled it restarted, etc., and that makes no difference.
Can you share the problem/error related to /etc/alertmanager.yml ?
@Naresh9999
Sorry, I’m not sure what you are asking. @Vadim_Yalovets first asked about the /etc/alertmanager.yml file, and the contents from above. That file does not look like what it should, I assume, and is not getting recreated properly upon restart.
Could you make sure that all pmm components have proper versions?
docker exec -it pmm-server bash -c 'yum list installed | egrep "pmm|percona"'
Looks like some of mine are slightly lower:
dbaas-controller.x86_64 2.25.0-1.2112011535.cd5e155.el7 @pmm2-server
percona-alertmanager.x86_64 0.21.0-3.el7 @pmm2-server
percona-dashboards.x86_64 2.25.0-15.2112081439.3c363b1.el7 @pmm2-server
percona-grafana.x86_64 7.5.11-94.2112031430.cfe5bd0.el7 @pmm2-server
percona-postgresql11.x86_64 1:11.9-2.el7 @percona-ppg-11
percona-postgresql11-contrib.x86_64
1:11.9-2.el7 @percona-ppg-11
percona-postgresql11-libs.x86_64 1:11.9-2.el7 @percona-ppg-11
percona-postgresql11-server.x86_64 1:11.9-2.el7 @percona-ppg-11
percona-prometheus.x86_64 2.21.0-1.el7 @local
percona-qan-api2.x86_64 2.25.0-15.2111040947.738aa30.el7 @pmm2-server
percona-release.noarch 1.0-26 @/percona-release-latest.noarchNUKY4L
percona-victoriametrics.x86_64 1.60.0-1.el7 @pmm2-server
pmm-managed.x86_64 2.25.0-17.2112130939.dc091ab.el7 @pmm2-server
pmm-server.noarch 2.25.0-25.2112130940.c22365c.el7 @pmm2-server
pmm-update.noarch 2.25.0-64.2112130941.7f24595.el7 @pmm2-server
pmm2-client.x86_64 2.25.0-6.el7 @percona-release-x86_64
Could you provide output of next commands?
amtool --version
amtool check-config /etc/alertmanager.yml
