I am attempting to figure out what I do not have correct here… I am using local storage for the screenshots and not a S3
I am able to do the test posts fine… Also able to get the Alerts into Slack fine… but No Images
Is this more or less due to using templates verse an actual Dashboard? Or is it for local storage they suggest the render plugin which is not able to be installed on the docker image as additional ldd files are required?
[unified_alerting.screenshots]
# Enable screenshots in notifications. This option requires the Grafana Image Renderer plugin.
# For more information on configuration options, refer to [rendering].
capture = true
render_timeout = 50s
concurent_render_limit = 130
max_concurrent_screenshots = 190
capture_timeout = 30s
upload_external_image_storage = true
# The maximum number of screenshots that can be taken at the same time. This option is different from
# concurrent_render_request_limit as max_concurrent_screenshots sets the number of concurrent screenshots
# that can be taken at the same time for all firing alerts where as concurrent_render_request_limit sets
# the total number of concurrent screenshots across all Grafana services.
max_concurrent_screenshots = 5
# Uploads screenshots to the local Grafana server or remote storage such as Azure, S3 and GCS. Please
# see [external_image_storage] for further configuration options. If this option is false then
# screenshots will be persisted to disk for up to temp_data_lifetime.
upload_external_image_storage = true
[external_image_storage]
# Used for uploading images to public servers so they can be included in slack/email messages.
# You can choose between (s3, webdav, gcs, azure_blob, local)
provider = local
capture = true
upload_external_image_storage = true
This option requires the Grafana Image Renderer plugin.
Have you installed this? If not, then the images won’t work.
@matthewb
I attempted to but had to remove it as grafana would not start up. I followed the instructions here
yum since it moved to 9 was not able to find some packages
o match for argument: libmodman
No match for argument: mailx
No match for argument: redhat-lsb-core
No match for argument: redhat-lsb-submod-security
No match for argument: spax
On Startup Grafana seems it could not load the pmm app aftewords…
logger=server t=2023-08-25T23:38:05.810892884Z level=error msg="Stopped background service" service=*rendering.RenderingService reason="Unrecognized remote plugin message: \n\nThis usually means that the plugin is either invalid or simply\nneeds to be recompiled to support the latest protocol."
logger=server t=2023-08-25T23:38:05.813533784Z level=error msg="Server shutdown" error="*rendering.RenderingService run error: Unrecognized remote plugin message: \n\nThis usually means that the plugin is either invalid or simply\nneeds to be recompiled to support the latest protocol."
*rendering.RenderingService run error: Unrecognized remote plugin message:
logger=plugindashboards t=2023-08-25T23:38:08.476487635Z level=error msg="Failed to load app dashboards" error="open /srv/grafana/plugins/pmm-app/dist/dashboards/Experimental/Environment_Summary.json: no such file or directory"
logger=server t=2023-08-25T23:38:09.482341768Z level=error msg="Stopped background service" service=*rendering.RenderingService reason="Unrecognized remote plugin message: \n\nThis usually means that the plugin is either invalid or simply\nneeds to be recompiled to support the latest protocol."
logger=server t=2023-08-25T23:38:09.485188003Z level=error msg="Server shutdown" error="*rendering.RenderingService run error: Unrecognized remote plugin message: \n\nThis usually means that the plugin is either invalid or simply\nneeds to be recompiled to support the latest protocol."
Once via the cli command was done and I removed the render plugin everything started to work again
After a bit of playing around it looks like 3.4.1 works anything newer seems to cause issues…
grafana-cli plugins install grafana-image-renderer 3.4.1 && supervisorctl restart grafana && tail -f /srv/logs/grafana.log
Can you please confirm what OS you are using? Also, are you running PMM via docker?
@matthewb
docker
Docker version 20.10.21, build baeda1f
pmm
Current version: 2.38.1 (July 11, 2023 UTC)
Fresh install a couple of weeks ago via the interactive pmm.sh script that checks/installs docker as needed.
The only changes I have made in the docker container are for
- setting http/https/no_proxy in the /etc/supervisord.conf (for Grafana to get to grafana.com behind our proxy setup)
environment=HTTP_PROXY=http://proxy.xxx.com:80
environment=HTTPS_PROXY=http://proxy.xxx.com:80
environment=http_proxy=http://proxy.xxx.com:80
environment=https_proxy=http://proxy.xxx.com:80
environment=no_proxy="*.xxxx.com,127.0.0.1"
environment=NO_PROXY="*.xxxx.com,127.0.0.1"
- enable ldap.toml file information to the domain controllers
- The above settings trying to get it to create the png file local vs using a S3 Bucket
Inside Docker
cat /etc/redhat-release
Red Hat Enterprise Linux release 9.2 (Plow)
Linux 7007a8dd1535 3.10.0-1160.92.1.el7.x86_64 #1 SMP Tue Jun 20 11:48:01 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Base OS
cat /etc/redhat-release
CentOS Linux release 7.9.2009 (Core)
uname -a
Linux xxxx 3.10.0-1160.92.1.el7.x86_64 #1 SMP Tue Jun 20 11:48:01 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
.
@matthewb did not know if any update existed on this or if I was doing items incorrectly?
These are the instructions we follow:
docker pull grafana/grafana-image-renderer:3.6.1
docker run -d --restart always -p 8081 --name renderer grafana/grafana-image-renderer:3.6.1
PMM_IP=$(docker inspect --format='{{.NetworkSettings.IPAddress}}' pmm-server)
RENDER_IP=$(docker inspect --format='{{.NetworkSettings.IPAddress}}' renderer)
echo "PMM Container: $PMM_IP | Grafana Renderer Container: $RENDER_IP"
docker exec \
-e RENDER_URL=http://${RENDER_IP}:8081/render \
-e CALLBACK=http://${PMM_IP}:3000/ \
pmm-server bash -c "echo -e \"
[rendering]\nserver_url=\${RENDER_URL}\ncallback_url=\${CALLBACK}\" >> /etc/grafana/grafana.ini"
docker exec pmm-server supervisorctl restart grafana
The above installs the render plugin as a separate docker container. Collects IP information. Appends a config section to grafana.ini, then restarts grafana. After that, images should now render in PMM.