Alerting and "Unexpected error" in graph rendered by GF Image Renderer service

[j_a_y]

Description:

I’m running two Docker containers, one for PMM and one for the Grafana Image Renderer service. I’ve used this link as a resource for this configuration:

Attaching a Percona Monitoring and Management Graph Image Along with an...

Regarding the alert notifications, although I’ve been able to successfully configure the environment so an image of the graph that corresponds with the alert is included in the email alert, the graph is incomplete. At the center of the graph panel it shows “No Data” and in the upper right-hand corner it shows “Unexpected Error”.

Steps to Reproduce:

[Step-by-step instructions on how to reproduce the issue, including any specific settings or configurations]

Version:

PMM: 2.44.0
GF Image Renderer: 3.12.1

Logs:

[If applicable, include any relevant log files or error messages]

Expected Result:

I’m expecting to see an image of a graph containing data relevant to the alert.

Actual Result:

An image of the graph is rendered but contains no data and an error message.

Additional Information:

I don’t know if the log messages are related but here are the docker logs for the “renderer” container when an alert is triggered:
docker logs [render_container_here]
{“level”:“error”,“message”:“Browser console error”,“msg”:“Failed to load resource: the server responded with a status of 401 ()”,“url”:“https://[server_name]/v1/inventory/Services/ListTypes”}
{“level”:“error”,“message”:“Browser console error”,“msg”:“Failed to load resource: the server responded with a status of 401 ()”,“url”:“https://[server_name]/v1/Settings/Get”}
{“level”:“error”,“message”:“Browser console error”,“msg”:“Failed to load resource: the server responded with a status of 401 ()”,“url”:“https://[server_name]/v1/Platform/UserStatus”}
{“level”:“error”,“message”:“Browser console error”,“msg”:“Failed to load resource: the server responded with a status of 401 ()”,“url”:“https://[server_name]/v1/management/Advisors/List”}
{“level”:“error”,“message”:“Browser console error”,“msg”:“Failed to load resource: the server responded with a status of 401 ()”,“url”:“https://[server_name]/v1/user”}
{“level”:“error”,“message”:“Browser console error”,“msg”:“Failed to load resource: the server responded with a status of 401 ()”,“url”:“https://[server_name]/v1/Platform/ServerInfo”}
{“level”:“error”,“message”:“Browser console error”,“msg”:“Failed to load resource: the server responded with a status of 401 ()”,“url”:“https://[server_name]/graph/api/ds/query”}

These 7 entries in the log file continuously repeat.

Thus far, I’ve not been able to find any information to help me understand the cause, or what’s missing in my configuration. Any thoughts, advice or info anyone has to share would be appreciated.

[matthewb]

401 mean unauthorized. Do you have roles/permissions enabled?

[matthewb]

Verify the graph is working. Navigate to that graph in PMM, then “Share” the graph and you should get an image link. Verify that image link shows the image.

[j_a_y]

Yes, when performing this action the image link does show the image I’d expect to see in the alert. It contains the graph and the data in the graph.

[j_a_y]

Hi Matthew, Perhaps this is the piece I’m missing? I’ve not configured anything specific to roles/permissions. Can you please point me in the right direction to see more details on this requirement? Thanks

[matthewb]

As far as I know you should not need any special permissions. Please show the docker commands you used to launch both containers.

[j_a_y]

services:
grafana-image-renderer-service:
environment:
- COMPOSE_PROJECT_NAME=pmm-project
- IGNORE_HTTPS_ERRORS=true
- GF_RENDERING_IGNORE_HTTPS_ERRORS=true
- ENABLE_METRICS=true
- GF_AUTH_TOKEN=12345
- GF_LOG_LEVEL=debug
- GF_LOG_FILTERS=rendering:debug
build:
context: ./renderer_build
dockerfile: Dockerfile
image: grafana/grafana-image-renderer:3.12.1
container_name: renderer
networks:
- pmm-network
restart: unless-stopped

pmm-service:
environment:
- COMPOSE_PROJECT_NAME=pmm-project
- GF_RENDERING_SERVER_URL=http://renderer:8081/render
- GF_RENDERING_CALLBACK_URL=https://[name-redacted]/graph/
- GF_RENDERER_TOKEN=12345
- GF_AUTH_ANONYMOUS_ENABLED=true
- GF_EXTERNAL_IMAGE_STORAGE_PROVIDER=local
- GF_ALERTING_ENABLED=false
- GF_UNIFIED_ALERTING_ENABLED=true
- GF_UNIFIED_ALERTING_SCREENSHOTS_CAPTURE=true
- GF_UNIFIED_ALERTING_SCREENSHOTS_CAPTURE_TIMEOUT=30s
- GF_UNIFIED_ALERTING_EVALUATION_TIMEOUT=90s
- GF_LOG_LEVEL=debug
- GF_LOG_FILTERS=rendering:debug
build:
context: ./pmm_build
dockerfile: Dockerfile
depends_on:
- grafana-image-renderer-service
ports:
- “443:443”
image: pmm-server:latest
container_name: pmm-server
networks:
- pmm-network
volumes:
- pmm-data:/srv
restart: unless-stopped
stdin_open: true
tty: true
command: /opt/entrypoint.sh

networks:
pmm-network:
name: pmm-network
external: true

volumes:
pmm-data:
name: pmm-data
external: true

[nurlan]

Can you try to clean browser cache and try it again?
What error message screenshot is containing?
Do you Access control configured for metrics?
Do you see anything else except those 401 errors?

[j_a_y]

Hi Nurlan,

The issue I’m experiencing doesn’t involve the use of a browser. Everything that involves the use of the browser with PMM is working well.

The issue is that embedded images in email alerts generated by PMM contain only an “empty” graph, along with an error message in the upper right hand corner of the graph (embedded image) that says “Unexpected error”. That’s it. As well, in the center of the graph image are the words “No data”.

So the images are being embedded in the email alerts, they are just not complete (they do not include the data expected in the graph).

When I go to the graph in question in PMM I can view the graph and the data no problem. I can also check to verify the image is properly rendered by “Sharing” the graph as matthewb mentioned previously.

In the Docker logs for the renderer container I see nothing else in the logs aside from the 401 errors shown above.

[j_a_y]

Sorry, should have specified…I’m using docker compose (command = docker compose up -d) and the docker-compose.yml file provided above.

[nurlan]

The reason why I asked you to clean cache it’s because grafana sends cookies from your browser to grafana image renderer which could lead to this problem you are facing. And since we changed cookies path during upgrade of PMM it could send outdated cookies.

[j_a_y]

Ok thanks for explaining. I did do what you asked, although still not understanding its relevance.

Sorry if I’m missing something, but let me explain a scenario that might better explain what I’m experiencing.

If I was only a recipient of email alerts from PMM and never opened the PMM site in my browser, my experience would be the following:

1. I would receive an alert sent to my email with the alert condition.
2. Upon opening the email I see a graph image in the body of the email but the graph contains “No data”. There are no lines on the chart indicating the metrics and the word “No data” are embedded in the center of the image. As well, in the upper right hand corner also embedded in the image are the words “Unexpected error”.
3. In this situation the use of a browser is not involved. I do not have to open a browser to view the email and the embedded image. I can view it in the email program.

Thanks for your reply.

[Yunus]

Hello Jay,
Using your docker compose file, we can configure the alert and get the dashboard as well.
Can you confirm if you have dashboard and your both docker instances can communicate.
You can use container names here instead like this,

      - GF_RENDERING_SERVER_URL=http://renderer:8081/render
      - GF_RENDERING_CALLBACK_URL=https://pmm-server:443/graph/

Have you configured any external authentication with grafana?
Can you try rendering the graph image from PMM itself
Here is an example when you view the graph and click share

Here we would like you to test if you click on “Direct link rendered image” do you see the graph?
Along with that if you can check grafana logs,
Exec into the pmm-server container and /srv/logs/grafana.log. Check if there are any errors there.
Let us know if that works.
Regards,
Yunus Shaikh.

[j_a_y]

I can open the dashboard and both docker instances can communicate with one another.

I have changed the names of my containers according to the names you have shown here, and that hasn’t changed anything.

I am using LDAP external authentication for accessing PMM.

I can render the graph image from PMM itself.

I am able to render an image by clicking on “Direct link rendered image”. The image includes the graph and the expected data.

I’ve also checked the grafana.log file during the firing of an alert and there are no errors logged. The only “errors” I see are from the renderer container when running docker logs command. Those are the errors initially reported above.

Here’s an example of the image I see in an email alert: