# AWS Marketplace PMM upgrade from 2.25.0 to 2.26.0 **URL:** \ **Category:** Percona Monitoring and Management (PMM)\ **Created:** [February 10, 2022, 7:54pm UTC](https://forums.percona.com/t/aws-marketplace-pmm-upgrade-from-2-25-0-to-2-26-0/14364 "2022-02-10T19:54:32Z")\ **Posts on this page:** 4\ **Page:** 2
**Author:** ![artem\_gavrilov](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/artem_gavrilov/32/1453_2.png) [@artem\_gavrilov](https://forums.percona.com/u/artem_gavrilov)\ **Post date:** [February 15, 2022, 4:38pm UTC](https://forums.percona.com/t/aws-marketplace-pmm-upgrade-from-2-25-0-to-2-26-0/14364/21 "2022-02-15T16:38:37Z")
After using default AM base config pmm-managed keeps restarting with the same error as before? ```auto ERRO[2022-02-11T14:09:43.585+00:00] Failed to update configuration, will retry: exit status 1 Checking ‘/tmp/pmm-managed-config-alertmanager-2940248744’ FAILED: yaml: unmarshal errors: line 51: field max_alerts not found in type config.plain amtool: error: failed to validate 1 file(s) ``` ---
**Author:** ![odemark1](https://avatars.discourse-cdn.com/v4/letter/o/82dd89/32.png) [@odemark1](https://forums.percona.com/u/odemark1)\ **Post date:** [February 16, 2022, 12:33pm UTC](https://forums.percona.com/t/aws-marketplace-pmm-upgrade-from-2-25-0-to-2-26-0/14364/22 "2022-02-16T12:33:12Z")
Yes, pmm-managed continue to restarts. It is happening continuously every few seconds. I never experience any issues with upgrade in the AWS Marketplace PMM. Upgrades happen monthly and the setup is similar to all PMM instances. Few of the instances (2 out of 9) are failing with similar issues. ---
**Author:** ![odemark1](https://avatars.discourse-cdn.com/v4/letter/o/82dd89/32.png) [@odemark1](https://forums.percona.com/u/odemark1)\ **Post date:** [February 17, 2022, 12:23pm UTC](https://forums.percona.com/t/aws-marketplace-pmm-upgrade-from-2-25-0-to-2-26-0/14364/23 "2022-02-17T12:23:51Z")
I do see a memory issue in pmm-managed. when checking pmm-managed.log, it has this statement below. panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x11afc0e] Here is the overall log. Alert Manager can be ignore as I have my own installed. This is case for all PMM servers and see the same thing for healthy environments. ``` /usr/local/go/src/runtime/asm_amd64.s:1581. component=setup ``` INFO[2022-02-17T12:22:25.615+00:00] prometheus configuration not changed, doing nothing. component=supervisord INFO[2022-02-17T12:22:25.615+00:00] victoriametrics configuration not changed, doing nothing. component=supervisord INFO[2022-02-17T12:22:25.615+00:00] vmalert configuration not changed, doing nothing. component=supervisord INFO[2022-02-17T12:22:25.615+00:00] alertmanager configuration not changed, doing nothing. component=supervisord INFO[2022-02-17T12:22:25.615+00:00] qan-api2 configuration not changed, doing nothing. component=supervisord INFO[2022-02-17T12:22:25.615+00:00] grafana configuration not changed, doing nothing. component=supervisord INFO[2022-02-17T12:22:25.615+00:00] dbaas-controller configuration not changed, doing nothing. component=supervisord INFO[2022-02-17T12:22:25.615+00:00] Checking VictoriaMetrics… component=setup INFO[2022-02-17T12:22:25.616+00:00] Checking VMAlert… component=setup INFO[2022-02-17T12:22:25.617+00:00] Checking Alertmanager… component=setup INFO[2022-02-17T12:22:25.617+00:00] Setup completed. component=setup INFO[2022-02-17T12:22:25.958+00:00] Starting services… component=main INFO[2022-02-17T12:22:25.958+00:00] Starting… component=victoriametrics INFO[2022-02-17T12:22:25.958+00:00] Starting server on [http://127.0.0.1:7771/](http://127.0.0.1:7771/) … component=gRPC INFO[2022-02-17T12:22:25.958+00:00] Starting… component=alertmanager INFO[2022-02-17T12:22:25.958+00:00] Starting… component=checks INFO[2022-02-17T12:22:25.958+00:00] Starting… component=vmalert INFO[2022-02-17T12:22:25.958+00:00] Using default SaaS host “[check.percona.com](http://check.percona.com)”. INFO[2022-02-17T12:22:25.958+00:00] Using SaaS host “[check.percona.com](http://check.percona.com)”. INFO[2022-02-17T12:22:25.958+00:00] Environment variable “PERCONA\_PLATFORM\_API\_TIMEOUT” is not set, using “30s” as a default timeout for platform API. component=auth INFO[2022-02-17T12:22:25.958+00:00] Starting server on [http://127.0.0.1:7772/](http://127.0.0.1:7772/) … component=JSON INFO[2022-02-17T12:22:25.958+00:00] Starting server on [http://127.0.0.1:7773/debug](http://127.0.0.1:7773/debug) Registered handlers: [http://127.0.0.1:7773/debug/metrics](http://127.0.0.1:7773/debug/metrics) [http://127.0.0.1:7773/debug/vars](http://127.0.0.1:7773/debug/vars) [http://127.0.0.1:7773/debug/requests](http://127.0.0.1:7773/debug/requests) [http://127.0.0.1:7773/debug/events](http://127.0.0.1:7773/debug/events) [http://127.0.0.1:7773/debug/pprof](http://127.0.0.1:7773/debug/pprof) component=debug INFO[2022-02-17T12:22:26.748+00:00] Starting Stream /agent.Agent/Connect … agent\_id=pmm-server request=4497a2aa-8fec-11ec-90b2-1260e5508cc3 INFO[2022-02-17T12:22:26.750+00:00] Connected pmm-agent: &{ID:pmm-server Version:2.26.0 MetricsPort:0}. agent\_id=pmm-server request=4497a2aa-8fec-11ec-90b2-1260e5508cc3 INFO[2022-02-17T12:22:26.750+00:00] Starting runStateChangeHandler … agent\_id=pmm-server request=4497a2aa-8fec-11ec-90b2-1260e5508cc3 INFO[2022-02-17T12:22:26.961+00:00] Configuration reloaded. component=vmalert INFO[2022-02-17T12:22:26.991+00:00] Configuration reloaded. component=alertmanager INFO[2022-02-17T12:22:27.138+00:00] Done. component=victoriametrics panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x11afc0e] goroutine 917 [running]: [github.com/percona/pmm-managed/services/victoriametrics.AddScrapeConfigs](http://github.com/percona/pmm-managed/services/victoriametrics.AddScrapeConfigs)(0xc00026c840, 0xc00026c840, 0x14ff92f, 0x8, 0x187b300, 0x0) /home/builder/rpm/BUILD/pmm-managed-6914083707b5478605e7bc816de8fc68b5f511f6/src/github.com/percona/pmm-managed/services/victoriametrics/prometheus.go:90 +0x68e [github.com/percona/pmm-managed/services/victoriametrics.(\*Service).populateConfig.func1(0xc000300078)](http://github.com/percona/pmm-managed/services/victoriametrics.(*Service).populateConfig.func1(0xc000300078)) /home/builder/rpm/BUILD/pmm-managed-6914083707b5478605e7bc816de8fc68b5f511f6/src/github.com/percona/pmm-managed/services/victoriametrics/victoriametrics.go:337 +0x625 gopkg.in/reform%2ev1.(\*DB).InTransactionContext(0xc001054b01, {0x18630c8, 0xc00003e038}, 0x1863138, 0xc000cb5d90) /home/builder/go/pkg/mod/gopkg.in/reform.v1@v1.5.1/db.go:93 +0xa2 gopkg.in/reform%2ev1.(\*DB).InTransaction(…) /home/builder/go/pkg/mod/gopkg.in/reform.v1@v1.5.1/db.go:74 [github.com/percona/pmm-managed/services/victoriametrics.(\*Service).populateConfig](http://github.com/percona/pmm-managed/services/victoriametrics.(*Service).populateConfig)(0xc000be70b0, 0xc07bad6179333954) /home/builder/rpm/BUILD/pmm-managed-6914083707b5478605e7bc816de8fc68b5f511f6/src/github.com/percona/pmm-managed/services/victoriametrics/victoriametrics.go:322 +0x56 [github.com/percona/pmm-managed/services/victoriametrics.(\*Service).marshalConfig](http://github.com/percona/pmm-managed/services/victoriametrics.(*Service).marshalConfig)(0xc000612140, 0x0) /home/builder/rpm/BUILD/pmm-managed-6914083707b5478605e7bc816de8fc68b5f511f6/src/github.com/percona/pmm-managed/services/victoriametrics/victoriametrics.go:215 +0x25 [github.com/percona/pmm-managed/services/victoriametrics.(\*Service).updateConfiguration](http://github.com/percona/pmm-managed/services/victoriametrics.(*Service).updateConfiguration)(0xc000612140, {0x1863100, 0xc001052360}) /home/builder/rpm/BUILD/pmm-managed-6914083707b5478605e7bc816de8fc68b5f511f6/src/github.com/percona/pmm-managed/services/victoriametrics/victoriametrics.go:157 +0xb5 [github.com/percona/pmm-managed/services/victoriametrics.(\*Service).Run](http://github.com/percona/pmm-managed/services/victoriametrics.(*Service).Run)(0xc000612140, {0x1863138, 0xc0005a8c30}) /home/builder/rpm/BUILD/pmm-managed-6914083707b5478605e7bc816de8fc68b5f511f6/src/github.com/percona/pmm-managed/services/victoriametrics/victoriametrics.go:130 +0x325 main.main.func6() /home/builder/rpm/BUILD/pmm-managed-6914083707b5478605e7bc816de8fc68b5f511f6/src/github.com/percona/pmm-managed/main.go:838 +0x65 created by main.main /home/builder/rpm/BUILD/pmm-managed-6914083707b5478605e7bc816de8fc68b5f511f6/src/github.com/percona/pmm-managed/main.go:836 +0x3fe5 ---
**Author:** ![odemark1](https://avatars.discourse-cdn.com/v4/letter/o/82dd89/32.png) [@odemark1](https://forums.percona.com/u/odemark1)\ **Post date:** [February 17, 2022, 4:45pm UTC](https://forums.percona.com/t/aws-marketplace-pmm-upgrade-from-2-25-0-to-2-26-0/14364/24 "2022-02-17T16:45:59Z")
AWS Marketplace PMM updates seem to be done by Ansible. What is the best way to see if process is complete? The pmm-update logs are not completing. It stops at time=“2022-02-17T16:32:25Z” level=info msg=“Waiting for Grafana dashboards update to finish…” in the logs. It always stops at this one. pmm-update-perform.log Starting “yum update --assumeyes pmm-update” … Loaded plugins: changelog, fastestmirror Loading mirror speeds from cached hostfile - elrepo: [dpep-centos-mirrors-latest-east.wdprapps.disney.com](http://dpep-centos-mirrors-latest-east.wdprapps.disney.com) - epel: [dl.fedoraproject.org](http://dl.fedoraproject.org) - fasttrack: [dpep-centos-mirrors-latest-east.wdprapps.disney.com](http://dpep-centos-mirrors-latest-east.wdprapps.disney.com) No packages marked for update Starting “yum --verbose info installed pmm-update” … Loading “changelog” plugin Loading “fastestmirror” plugin Config time: 0.010 Yum version: 3.4.3 rpmdb time: 0.000 Installed Packages Name : pmm-update Arch : noarch Version : 2.26.0 Release : 64.2201121502.f13eec0.el7 Size : 2.2 M Repo : installed From repo : pmm2-server Committer : Nikita Beletskii [nikita.beletskii@percona.com](mailto:nikita.beletskii@percona.com) Committime : Tue Oct 19 12:00:00 2021 Buildtime : Wed Jan 12 15:02:19 2022 Install time: Thu Feb 10 10:57:14 2022 Installed by: 1000 Changed by : System Summary : Tool for updating packages and OS configuration for PMM Server URL : [GitHub - percona/pmm-update: PMM Update is a tool for updating packages and OS configuration for Percona Monitoring and Management Server.](https://github.com/percona/pmm-update) License : AGPLv3 Description : Tool for updating packages and OS configuration for PMM Server time=“2022-02-17T16:31:49Z” level=info msg=“pmm-update:\nbefore update = {Version:2.26.0 FullVersion:2.26.0-64.2201121502.f13eec0.el7 BuildTime:2022-01-12 15:02:19 +0000 UTC Repo:pmm2-server}\n after update = {Version:2.26.0 FullVersion:2.26.0-64.2201121502.f13eec0.el7 BuildTime:2022-01-12 15:02:19 +0000 UTC Repo:pmm2-server}” time=“2022-02-17T16:31:49Z” level=info msg=“Version did not change.” Starting “ansible-playbook --flush-cache /usr/share/pmm-update/ansible/playbook/tasks/update.yml” … [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match ‘all’ PLAY [localhost] \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\* TASK [Gathering Facts] \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\* ok: [localhost] TASK [detect /srv/pmm-distribution] \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\* ok: [localhost] TASK [detect containers] \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\* ok: [localhost] TASK [force container] \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\* skipping: [localhost] TASK [Remove percona-dashboard without architecture] \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\* ok: [localhost] TASK [Update percona-dashboards package] \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\* ok: [localhost] TASK [Upgrade grafana database (New schema)] \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\* changed: [localhost] TASK [Create provisioning directory] \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\* ok: [localhost] =\> (item=datasources) ok: [localhost] =\> (item=plugins) ok: [localhost] =\> (item=dashboards) TASK [Copy grafana provisioning files] \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\* ok: [localhost] =\> (item=datasources) ok: [localhost] =\> (item=plugins) ok: [localhost] =\> (item=dashboards) TASK [Supervisord start | Start supervisord service for AMI/OVF] \*\*\*\*\*\*\*\* ok: [localhost] TASK [Check that supervisor socket exists] \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\* ok: [localhost] TASK [Supervisord start | Start supervisord for docker] \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\* skipping: [localhost] TASK [Run initialization playbook] \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\* TASK [initialization : Get current version] \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\* ok: [localhost] TASK [initialization : Get image version] \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\* ok: [localhost] TASK [initialization : Set current version if VERSION fail doen’t exist] \*\*\*\*\*\*\* skipping: [localhost] TASK [initialization : Setting current and image version] \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\* ok: [localhost] TASK [initialization : Setting current and image version] \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\* ok: [localhost] TASK [initialization : Print current version] \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\* ok: [localhost] =\> { “msg”: “Current version: 2.26.0 Image Version: 2.25.0” } PLAY RECAP \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\* localhost : ok=15 changed=1 unreachable=0 failed=0 skipped=3 rescued=0 ignored=0 time=“2022-02-17T16:32:25Z” level=info msg=“Waiting for Grafana dashboards update to finish…” [Previous page](https://forums.percona.com/t/aws-marketplace-pmm-upgrade-from-2-25-0-to-2-26-0/14364.md?page=1)