I’ve updated to the latest 2.25 version, and I’m seeing these messages scroll over and over in /srv/logs/pmm-managed.log
WARN[2022-01-14T15:55:48.151+00:00] Setting global email config, any user defined changes to the base config might be overwritten. component=alertmanager
ERRO[2022-01-14T15:55:48.182+00:00] Checking '/tmp/pmm-managed-config-alertmanager-076327072' FAILED: address 127.0.0.1: missing port in address
amtool: error: failed to validate 1 file(s)
component=alertmanager
ERRO[2022-01-14T15:55:48.182+00:00] Failed to update configuration, will retry: exit status 1
Checking '/tmp/pmm-managed-config-alertmanager-076327072' FAILED: address 127.0.0.1: missing port in address
amtool: error: failed to validate 1 file(s)
github.com/percona/pmm-managed/services/alertmanager.(*Service).validateConfig
/home/builder/rpm/BUILD/pmm-managed-dc091abb30f1e0ac8769a02d0587aa1be653a58c/src/github.com/percona/pmm-managed/services/alertmanager/alertmanager.go:292
github.com/percona/pmm-managed/services/alertmanager.(*Service).configAndReload
/home/builder/rpm/BUILD/pmm-managed-dc091abb30f1e0ac8769a02d0587aa1be653a58c/src/github.com/percona/pmm-managed/services/alertmanager/alertmanager.go:325
github.com/percona/pmm-managed/services/alertmanager.(*Service).updateConfiguration
/home/builder/rpm/BUILD/pmm-managed-dc091abb30f1e0ac8769a02d0587aa1be653a58c/src/github.com/percona/pmm-managed/services/alertmanager/alertmanager.go:205
github.com/percona/pmm-managed/services/alertmanager.(*Service).Run
/home/builder/rpm/BUILD/pmm-managed-dc091abb30f1e0ac8769a02d0587aa1be653a58c/src/github.com/percona/pmm-managed/services/alertmanager/alertmanager.go:173
main.main.func7
/home/builder/rpm/BUILD/pmm-managed-dc091abb30f1e0ac8769a02d0587aa1be653a58c/src/github.com/percona/pmm-managed/main.go:841
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1371. component=alertmanager
I do not have “Integrated Alerting” enabled, and I’m not sure where this is coming from. Any suggestions?
Hi Danny,
Could check any errors in next log files?
/srv/logs/alertmanager.log
/srv/logs/pmm-update-perform.log
1 Like
Hey Vadim.
Here is: /srv/logs/alertmanager.log
level=info ts=2022-01-14T15:47:44.638Z caller=main.go:524 msg="Received SIGTERM, exiting gracefully..."
level=info ts=2022-01-14T15:47:48.627Z caller=main.go:216 msg="Starting Alertmanager" version="(version=0.21.0, branch=non-git, revision=non-git)"
level=info ts=2022-01-14T15:47:48.627Z caller=main.go:217 build_context="(go=go1.15.7, user=builder@b33c58619690, date=20210517-08:24:43)"
level=info ts=2022-01-14T15:47:48.671Z caller=coordinator.go:119 component=configuration msg="Loading configuration file" file=/etc/alertmanager.yml
level=info ts=2022-01-14T15:47:48.671Z caller=coordinator.go:131 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager.yml
level=info ts=2022-01-14T15:47:48.674Z caller=main.go:485 msg=Listening address=127.0.0.1:9093
Contents of /etc/alertmanager.yml
---
# You can edit this file; changes will be preserved.
route:
receiver: empty
routes: []
receivers:
- name: empty
No errors in /srv/logs/pmm-update-perform.log that I can see.
I’m assuming Alertmanager and Integrated Alerting are two separate things, so with Integrated Alerting shut off it makes no difference. However, I didn’t configure anything specific for Alertmanager, so not sure why that would be having issues.
1 Like
Hi @dbray_sd
Did you modify the below file, can you please check once in the below file?
cat /etc/grafana/grafana.ini
May be this might be the reason, I am not sure but just try to check this once.
1 Like
It looks like the configuration files is protected for writing.
So pmm-managed service can’t recreate it.
e.g.
Could you remove this file and restart pmm-managed service?
2 Likes
Hey @Naresh9999
No, I did not manually modify that file.
[smtp]
;enabled = false
;host = localhost:25
;user =
# If the password contains # or ; you have to wrap it with triple quotes. Ex """#password;"""
;password =
;cert_file =
;key_file =
;skip_verify = false
;from_address = admin@grafana.localhost
;from_name = Grafana
# EHLO identity in SMTP dialog (defaults to instance_name)
;ehlo_identity = dashboard.example.com
# SMTP startTLS policy (defaults to 'OpportunisticStartTLS')
;startTLS_policy = NoStartTLS
2 Likes
@Vadim_Yalovets
Odd, it recreated it…but recreated it with the exact same contents:
[root@1fd26d7c5d00 opt]# supervisorctl stop pmm-managed
pmm-managed: stopped
[root@1fd26d7c5d00 opt]# rm /etc/alertmanager.yml
rm: remove regular file ‘/etc/alertmanager.yml’? y
[root@1fd26d7c5d00 opt]# supervisorctl start pmm-managed
pmm-managed: started
[root@1fd26d7c5d00 opt]# cat /etc/alertmanager.yml
---
# You can edit this file; changes will be preserved.
route:
receiver: empty
routes: []
receivers:
- name: empty
1 Like
Hi @dbray_sd
Can you stop and start the docker?
1 Like
@Naresh9999
Already tried that. It still recreates the file with the same “empty” contents as above.
Thanks.
1 Like
Hi @dbray_sd
Is the below file still exists with the same name?
/tmp/pmm-managed-config-alertmanager-076327072
can you remove it and then restart the pmm server?
1 Like
My file was named:
/tmp/pmm-managed-config-alertmanager-087463742
…I removed it and /etc/alertmanager.yml restarted. Problem remains, /etc/alertmanager.yml recreated with same empty contents, and no new /tmp/pmm-managed-config-alertmanager-* file was created.
1 Like
Did you enable the Alerting in the GUI before upgrade to 2.25?
1 Like
If you mean the “Integrated Alerting”, yes, that was enabled, but was never used/configured. So it was disabled after upgrading to 2.25. However, this issue was happening before the upgrade to 2.25.
1 Like
Can you disable(in gui) it now and then restart the pmm server?
1 Like
Already tried that as well. It has been disabled this entire time in the gui, before I posted to this forum. I’ve also enabled it restarted, disabled it restarted, etc., and that makes no difference.
1 Like
Can you share the problem/error related to /etc/alertmanager.yml ?
1 Like
@Naresh9999
Sorry, I’m not sure what you are asking. @Vadim_Yalovets first asked about the /etc/alertmanager.yml file, and the contents from above. That file does not look like what it should, I assume, and is not getting recreated properly upon restart.
1 Like
Could you make sure that all pmm components have proper versions?
docker exec -it pmm-server bash -c 'yum list installed | egrep "pmm|percona"'
1 Like
Looks like some of mine are slightly lower:
dbaas-controller.x86_64 2.25.0-1.2112011535.cd5e155.el7 @pmm2-server
percona-alertmanager.x86_64 0.21.0-3.el7 @pmm2-server
percona-dashboards.x86_64 2.25.0-15.2112081439.3c363b1.el7 @pmm2-server
percona-grafana.x86_64 7.5.11-94.2112031430.cfe5bd0.el7 @pmm2-server
percona-postgresql11.x86_64 1:11.9-2.el7 @percona-ppg-11
percona-postgresql11-contrib.x86_64
1:11.9-2.el7 @percona-ppg-11
percona-postgresql11-libs.x86_64 1:11.9-2.el7 @percona-ppg-11
percona-postgresql11-server.x86_64 1:11.9-2.el7 @percona-ppg-11
percona-prometheus.x86_64 2.21.0-1.el7 @local
percona-qan-api2.x86_64 2.25.0-15.2111040947.738aa30.el7 @pmm2-server
percona-release.noarch 1.0-26 @/percona-release-latest.noarchNUKY4L
percona-victoriametrics.x86_64 1.60.0-1.el7 @pmm2-server
pmm-managed.x86_64 2.25.0-17.2112130939.dc091ab.el7 @pmm2-server
pmm-server.noarch 2.25.0-25.2112130940.c22365c.el7 @pmm2-server
pmm-update.noarch 2.25.0-64.2112130941.7f24595.el7 @pmm2-server
pmm2-client.x86_64 2.25.0-6.el7 @percona-release-x86_64
1 Like
Could you provide output of next commands?
amtool --version
amtool check-config /etc/alertmanager.yml
1 Like
