AlertManager error scrolling over and over in /srv/logs/pmm-managed.log

I’ve updated to the latest 2.25 version, and I’m seeing these messages scroll over and over in /srv/logs/pmm-managed.log

WARN[2022-01-14T15:55:48.151+00:00] Setting global email config, any user defined changes to the base config might be overwritten.  component=alertmanager
ERRO[2022-01-14T15:55:48.182+00:00] Checking '/tmp/pmm-managed-config-alertmanager-076327072'  FAILED: address 127.0.0.1: missing port in address

amtool: error: failed to validate 1 file(s)
  component=alertmanager
ERRO[2022-01-14T15:55:48.182+00:00] Failed to update configuration, will retry: exit status 1
Checking '/tmp/pmm-managed-config-alertmanager-076327072'  FAILED: address 127.0.0.1: missing port in address

amtool: error: failed to validate 1 file(s)


github.com/percona/pmm-managed/services/alertmanager.(*Service).validateConfig
        /home/builder/rpm/BUILD/pmm-managed-dc091abb30f1e0ac8769a02d0587aa1be653a58c/src/github.com/percona/pmm-managed/services/alertmanager/alertmanager.go:292
github.com/percona/pmm-managed/services/alertmanager.(*Service).configAndReload
        /home/builder/rpm/BUILD/pmm-managed-dc091abb30f1e0ac8769a02d0587aa1be653a58c/src/github.com/percona/pmm-managed/services/alertmanager/alertmanager.go:325
github.com/percona/pmm-managed/services/alertmanager.(*Service).updateConfiguration
        /home/builder/rpm/BUILD/pmm-managed-dc091abb30f1e0ac8769a02d0587aa1be653a58c/src/github.com/percona/pmm-managed/services/alertmanager/alertmanager.go:205
github.com/percona/pmm-managed/services/alertmanager.(*Service).Run
        /home/builder/rpm/BUILD/pmm-managed-dc091abb30f1e0ac8769a02d0587aa1be653a58c/src/github.com/percona/pmm-managed/services/alertmanager/alertmanager.go:173
main.main.func7
        /home/builder/rpm/BUILD/pmm-managed-dc091abb30f1e0ac8769a02d0587aa1be653a58c/src/github.com/percona/pmm-managed/main.go:841
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1371.  component=alertmanager

I do not have “Integrated Alerting” enabled, and I’m not sure where this is coming from. Any suggestions?

Hi Danny,

Could check any errors in next log files?

/srv/logs/alertmanager.log
/srv/logs/pmm-update-perform.log

1 Like

Hey Vadim.

Here is: /srv/logs/alertmanager.log

level=info ts=2022-01-14T15:47:44.638Z caller=main.go:524 msg="Received SIGTERM, exiting gracefully..."
level=info ts=2022-01-14T15:47:48.627Z caller=main.go:216 msg="Starting Alertmanager" version="(version=0.21.0, branch=non-git, revision=non-git)"
level=info ts=2022-01-14T15:47:48.627Z caller=main.go:217 build_context="(go=go1.15.7, user=builder@b33c58619690, date=20210517-08:24:43)"
level=info ts=2022-01-14T15:47:48.671Z caller=coordinator.go:119 component=configuration msg="Loading configuration file" file=/etc/alertmanager.yml
level=info ts=2022-01-14T15:47:48.671Z caller=coordinator.go:131 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager.yml
level=info ts=2022-01-14T15:47:48.674Z caller=main.go:485 msg=Listening address=127.0.0.1:9093

Contents of /etc/alertmanager.yml

---
# You can edit this file; changes will be preserved.

route:
    receiver: empty
    routes: []

receivers:
    - name: empty

No errors in /srv/logs/pmm-update-perform.log that I can see.

I’m assuming Alertmanager and Integrated Alerting are two separate things, so with Integrated Alerting shut off it makes no difference. However, I didn’t configure anything specific for Alertmanager, so not sure why that would be having issues.

1 Like

Hi @dbray_sd

Did you modify the below file, can you please check once in the below file?
cat /etc/grafana/grafana.ini

May be this might be the reason, I am not sure but just try to check this once.

1 Like

It looks like the configuration files is protected for writing.
So pmm-managed service can’t recreate it.

e.g.
Screenshot 2022-01-15 at 14.56.30

Could you remove this file and restart pmm-managed service?

2 Likes

Hey @Naresh9999

No, I did not manually modify that file.

[smtp]
;enabled = false
;host = localhost:25
;user =
# If the password contains # or ; you have to wrap it with triple quotes. Ex """#password;"""
;password =
;cert_file =
;key_file =
;skip_verify = false
;from_address = admin@grafana.localhost
;from_name = Grafana
# EHLO identity in SMTP dialog (defaults to instance_name)
;ehlo_identity = dashboard.example.com
# SMTP startTLS policy (defaults to 'OpportunisticStartTLS')
;startTLS_policy = NoStartTLS

2 Likes

@Vadim_Yalovets

Odd, it recreated it…but recreated it with the exact same contents:

[root@1fd26d7c5d00 opt]# supervisorctl stop pmm-managed
pmm-managed: stopped
[root@1fd26d7c5d00 opt]# rm /etc/alertmanager.yml
rm: remove regular file ‘/etc/alertmanager.yml’? y
[root@1fd26d7c5d00 opt]# supervisorctl start pmm-managed
pmm-managed: started
[root@1fd26d7c5d00 opt]# cat /etc/alertmanager.yml
---
# You can edit this file; changes will be preserved.

route:
    receiver: empty
    routes: []

receivers:
    - name: empty

1 Like

Hi @dbray_sd

Can you stop and start the docker?

1 Like

@Naresh9999

Already tried that. It still recreates the file with the same “empty” contents as above.

Thanks.

1 Like

Hi @dbray_sd

Is the below file still exists with the same name?
/tmp/pmm-managed-config-alertmanager-076327072

can you remove it and then restart the pmm server?

1 Like

My file was named:
/tmp/pmm-managed-config-alertmanager-087463742

…I removed it and /etc/alertmanager.yml restarted. Problem remains, /etc/alertmanager.yml recreated with same empty contents, and no new /tmp/pmm-managed-config-alertmanager-* file was created.

1 Like

Did you enable the Alerting in the GUI before upgrade to 2.25?

1 Like

If you mean the “Integrated Alerting”, yes, that was enabled, but was never used/configured. So it was disabled after upgrading to 2.25. However, this issue was happening before the upgrade to 2.25.

1 Like

Can you disable(in gui) it now and then restart the pmm server?

1 Like

Already tried that as well. It has been disabled this entire time in the gui, before I posted to this forum. I’ve also enabled it restarted, disabled it restarted, etc., and that makes no difference.

1 Like

Can you share the problem/error related to /etc/alertmanager.yml ?

1 Like

@Naresh9999
Sorry, I’m not sure what you are asking. @Vadim_Yalovets first asked about the /etc/alertmanager.yml file, and the contents from above. That file does not look like what it should, I assume, and is not getting recreated properly upon restart.

1 Like

Could you make sure that all pmm components have proper versions?

docker exec -it pmm-server bash -c 'yum list installed | egrep "pmm|percona"'

1 Like

Looks like some of mine are slightly lower:

dbaas-controller.x86_64            2.25.0-1.2112011535.cd5e155.el7  @pmm2-server
percona-alertmanager.x86_64        0.21.0-3.el7                     @pmm2-server
percona-dashboards.x86_64          2.25.0-15.2112081439.3c363b1.el7 @pmm2-server
percona-grafana.x86_64             7.5.11-94.2112031430.cfe5bd0.el7 @pmm2-server
percona-postgresql11.x86_64        1:11.9-2.el7                     @percona-ppg-11
percona-postgresql11-contrib.x86_64
                                   1:11.9-2.el7                     @percona-ppg-11
percona-postgresql11-libs.x86_64   1:11.9-2.el7                     @percona-ppg-11
percona-postgresql11-server.x86_64 1:11.9-2.el7                     @percona-ppg-11
percona-prometheus.x86_64          2.21.0-1.el7                     @local
percona-qan-api2.x86_64            2.25.0-15.2111040947.738aa30.el7 @pmm2-server
percona-release.noarch             1.0-26                           @/percona-release-latest.noarchNUKY4L
percona-victoriametrics.x86_64     1.60.0-1.el7                     @pmm2-server
pmm-managed.x86_64                 2.25.0-17.2112130939.dc091ab.el7 @pmm2-server
pmm-server.noarch                  2.25.0-25.2112130940.c22365c.el7 @pmm2-server
pmm-update.noarch                  2.25.0-64.2112130941.7f24595.el7 @pmm2-server
pmm2-client.x86_64                 2.25.0-6.el7                     @percona-release-x86_64

1 Like

Could you provide output of next commands?

amtool --version
amtool check-config /etc/alertmanager.yml

Screenshot 2022-01-17 at 18.50.02

1 Like