Dashboard briefly connects and then EC2 server becomes inaccessible

Percona Monitoring and Management (PMM) Database Monitoring and Management
1

Hi,

I followed this documentation exactly with Instance type: m5.large. My EBS volumes provide 8GB for the boot volume and 100GB for the configuration. AWS Marketplace - Percona Monitoring and Management

I can briefly log into the dashboard for a few minutes. Then I get this error: “Unexpected error” within the dashboard. I can only reconnect to the server after an instance reboot. But the error keeps on happening within a few minutes.

Thanks,
Mark

1 Like
2

Hi, I suggest you try connecting to the PMM instance over SSH and check inside /srv/logs for clues.

1 Like
3

Thanks, I will have a look.

1 Like
4

I wonder if this could be the issue from pmm-managed.log?

Environment variable “PERCONA_PLATFORM_API_TIMEOUT” is not set, using “30s” as a default timeout for platform API. ^[[36mcomponent^[[0m=“portal client”

1 Like
5

Or this one from pmm-agent.log:

^[[36mINFO^[[0m[2022-11-03T04:02:13.929+00:00] Connecting to https://127.0.0.1:443/ … ^[[36mcomponent^[[0m=client
^[[36mINFO^[[0m[2022-11-03T04:02:13.942+00:00] Connected to 127.0.0.1:443. ^[[36mcomponent^[[0m=client
^[[36mINFO^[[0m[2022-11-03T04:02:13.942+00:00] Establishing two-way communication channel … ^[[36mcomponent^[[0m=client
^[[31mERRO^[[0m[2022-11-03T04:02:13.944+00:00] Failed to establish two-way communication channel: unexpected HTTP status code received from server: 500 (Internal Server Error); transport: received unexpected content-type “text/html”. ^[[31mcomponent^[[0m=client

1 Like
6

alertmanager.log:

[admin@ip-172-31-18-203 logs]$ vi alertmanager.log
level=info ts=2022-11-03T04:02:09.359Z caller=main.go:221 msg=“Starting Alertmanager” version=“(version=0.22.0, branch=non-git, revision=non-git)”
level=info ts=2022-11-03T04:02:09.359Z caller=main.go:222 build_context=“(go=go1.18.4, user=builder@b3c0dcc1e11b, date=20220905-08:47:10)”
level=info ts=2022-11-03T04:02:09.439Z caller=coordinator.go:113 component=configuration msg=“Loading configuration file” file=/etc/alertmanager.yml
level=info ts=2022-11-03T04:02:09.447Z caller=coordinator.go:126 component=configuration msg=“Completed loading of configuration file” file=/etc/alertmanager.yml
level=info ts=2022-11-03T04:02:09.450Z caller=main.go:414 component=configuration msg=“skipping creation of receiver not referenced by any route” receiver=disabled
level=info ts=2022-11-03T04:02:09.451Z caller=main.go:514 msg=Listening address=127.0.0.1:9093
level=info ts=2022-11-03T04:02:09.452Z caller=tls_config.go:191 msg=“TLS is disabled.” http2=false
level=info ts=2022-11-03T04:02:17.536Z caller=coordinator.go:113 component=configuration msg=“Loading configuration file” file=/etc/alertmanager.yml
level=info ts=2022-11-03T04:02:17.536Z caller=coordinator.go:126 component=configuration msg=“Completed loading of configuration file” file=/etc/alertmanager.yml
level=info ts=2022-11-03T04:02:17.539Z caller=main.go:414 component=configuration msg=“skipping creation of receiver not referenced by any route” receiver=disabled
level=info ts=2022-11-03T04:03:15.799Z caller=main.go:553 msg=“Received SIGTERM, exiting gracefully…”

1 Like
7

Hello, the above output does not tell me much. The default timeout should be enough… I suggest you open a bug in jira.percona.com so the dev team can take a look.

1 Like
8

Thanks. I downgraded to the previous version of Percona (software version: 2.30.0) and that appears to have resolved the issue. I will put in a bug ticket.

1 Like