Alerting and "Unexpected error" in graph rendered by GF Image Renderer service

Percona Monitoring and Management (PMM) PMM 2.x
1

Description:

I’m running two Docker containers, one for PMM and one for the Grafana Image Renderer service. I’ve used this link as a resource for this configuration:

Regarding the alert notifications, although I’ve been able to successfully configure the environment so an image of the graph that corresponds with the alert is included in the email alert, the graph is incomplete. At the center of the graph panel it shows “No Data” and in the upper right-hand corner it shows “Unexpected Error”.

Steps to Reproduce:

[Step-by-step instructions on how to reproduce the issue, including any specific settings or configurations]

Version:

PMM: 2.44.0
GF Image Renderer: 3.12.1

Logs:

[If applicable, include any relevant log files or error messages]

Expected Result:

I’m expecting to see an image of a graph containing data relevant to the alert.

Actual Result:

An image of the graph is rendered but contains no data and an error message.

Additional Information:

I don’t know if the log messages are related but here are the docker logs for the “renderer” container when an alert is triggered:
docker logs [render_container_here]
{“level”:“error”,“message”:“Browser console error”,“msg”:“Failed to load resource: the server responded with a status of 401 ()”,“url”:“https://[server_name]/v1/inventory/Services/ListTypes”}
{“level”:“error”,“message”:“Browser console error”,“msg”:“Failed to load resource: the server responded with a status of 401 ()”,“url”:“https://[server_name]/v1/Settings/Get”}
{“level”:“error”,“message”:“Browser console error”,“msg”:“Failed to load resource: the server responded with a status of 401 ()”,“url”:“https://[server_name]/v1/Platform/UserStatus”}
{“level”:“error”,“message”:“Browser console error”,“msg”:“Failed to load resource: the server responded with a status of 401 ()”,“url”:“https://[server_name]/v1/management/Advisors/List”}
{“level”:“error”,“message”:“Browser console error”,“msg”:“Failed to load resource: the server responded with a status of 401 ()”,“url”:“https://[server_name]/v1/user”}
{“level”:“error”,“message”:“Browser console error”,“msg”:“Failed to load resource: the server responded with a status of 401 ()”,“url”:“https://[server_name]/v1/Platform/ServerInfo”}
{“level”:“error”,“message”:“Browser console error”,“msg”:“Failed to load resource: the server responded with a status of 401 ()”,“url”:“https://[server_name]/graph/api/ds/query”}

These 7 entries in the log file continuously repeat.

Thus far, I’ve not been able to find any information to help me understand the cause, or what’s missing in my configuration. Any thoughts, advice or info anyone has to share would be appreciated.

2

401 mean unauthorized. Do you have roles/permissions enabled?

3

Verify the graph is working. Navigate to that graph in PMM, then “Share” the graph and you should get an image link. Verify that image link shows the image.

4

Yes, when performing this action the image link does show the image I’d expect to see in the alert. It contains the graph and the data in the graph.

5

Hi Matthew, Perhaps this is the piece I’m missing? I’ve not configured anything specific to roles/permissions. Can you please point me in the right direction to see more details on this requirement? Thanks

6

As far as I know you should not need any special permissions. Please show the docker commands you used to launch both containers.

7

services:
grafana-image-renderer-service:
environment:
- COMPOSE_PROJECT_NAME=pmm-project
- IGNORE_HTTPS_ERRORS=true
- GF_RENDERING_IGNORE_HTTPS_ERRORS=true
- ENABLE_METRICS=true
- GF_AUTH_TOKEN=12345
- GF_LOG_LEVEL=debug
- GF_LOG_FILTERS=rendering:debug
build:
context: ./renderer_build
dockerfile: Dockerfile
image: grafana/grafana-image-renderer:3.12.1
container_name: renderer
networks:
- pmm-network
restart: unless-stopped

pmm-service:
environment:
- COMPOSE_PROJECT_NAME=pmm-project
- GF_RENDERING_SERVER_URL=http://renderer:8081/render
- GF_RENDERING_CALLBACK_URL=https://[name-redacted]/graph/
- GF_RENDERER_TOKEN=12345
- GF_AUTH_ANONYMOUS_ENABLED=true
- GF_EXTERNAL_IMAGE_STORAGE_PROVIDER=local
- GF_ALERTING_ENABLED=false
- GF_UNIFIED_ALERTING_ENABLED=true
- GF_UNIFIED_ALERTING_SCREENSHOTS_CAPTURE=true
- GF_UNIFIED_ALERTING_SCREENSHOTS_CAPTURE_TIMEOUT=30s
- GF_UNIFIED_ALERTING_EVALUATION_TIMEOUT=90s
- GF_LOG_LEVEL=debug
- GF_LOG_FILTERS=rendering:debug
build:
context: ./pmm_build
dockerfile: Dockerfile
depends_on:
- grafana-image-renderer-service
ports:
- “443:443”
image: pmm-server:latest
container_name: pmm-server
networks:
- pmm-network
volumes:
- pmm-data:/srv
restart: unless-stopped
stdin_open: true
tty: true
command: /opt/entrypoint.sh

networks:
pmm-network:
name: pmm-network
external: true

volumes:
pmm-data:
name: pmm-data
external: true

8

Can you try to clean browser cache and try it again?
What error message screenshot is containing?
Do you Access control configured for metrics?
Do you see anything else except those 401 errors?

9

Hi Nurlan,

The issue I’m experiencing doesn’t involve the use of a browser. Everything that involves the use of the browser with PMM is working well.

The issue is that embedded images in email alerts generated by PMM contain only an “empty” graph, along with an error message in the upper right hand corner of the graph (embedded image) that says “Unexpected error”. That’s it. As well, in the center of the graph image are the words “No data”.

So the images are being embedded in the email alerts, they are just not complete (they do not include the data expected in the graph).

When I go to the graph in question in PMM I can view the graph and the data no problem. I can also check to verify the image is properly rendered by “Sharing” the graph as matthewb mentioned previously.

In the Docker logs for the renderer container I see nothing else in the logs aside from the 401 errors shown above.

10

Sorry, should have specified…I’m using docker compose (command = docker compose up -d) and the docker-compose.yml file provided above.