Unknown panel type: pmm-singlestat-panel and missing dashboards in v1.13.0


In the process of upgrading my PMM docker instance on the production server, I’ve made a type that probably caused (or at least contributed to) all the grief and confusion I’m suffering.

After stopping the v1.12.0 container, renaming it, and pulling the v1.13.0 image, I’ve started the container using ‘percona/pmm-server:1.1.0’ as the image name. There was no such image on the server, in which case according to the manual, the docker would pull the missing image. The image download process didn’t show on the command line (like when you run ‘docker pull’), so I didn’t spot the typo until later, and I’m unsure if this is the cause.
On startup, loading of the “CPU Busy” panels under the “Environment overview” would time-out and throw an error, while the server would choke because its IOPS capacity was being maxed out (all 4 cores reported >70% iowait). Later on, I noticed one of the two prometheus processes would eventually get killed by oom-killer just after its memory consumption rapidly reached 70% (out of 8GB). I’ve removed the ‘bad’ v1.1.0 container and started a new one specifying the correct image.

Strangely, by the next day the server, and PMM running on it got stabilized but now some things were missing - the PMM App somehow got disabled!
After enabling the PMM app/plugin, I noticed that in addition to the singlestat-panel problem, at least the following dashboards were missing: Network Overview, Overview NUMA metrics,… while the “Home Dashboard” displays as on the screenshot attached when loaded via http:///

Removing the container, and re-creating it doesn’t fix the missing dashboards and panel.

I suspect the v1.13.0 image got somehow corrupt, and am considering removal and re-download, but because Docker is just a tool I’m not too familiar with,
I’d appreciate some guidance on how to fix this as painlessly as possible.


Just tried removing the v1.13 image, and downloading it from the repository again, but no dice.

Could you provide the contents of the /var/log/dashboard-upgrade.log file please. Or if you don’t have that, could you please run the import dashboards script manually - and share output with us. Thanks.

Hi Lorraine, the best I can do is the ‘screen’ output when I glanced at the log myself. See attached.
I’m afraid the log itself got nuked when I removed the v1.13.0 image and pulled back from the repo.

A silly question: Where is the binary I would run so as to import the dashboards manually?

On a side note: the two missing dashboards (Overview NUMA metrics and Network Overview) I’ve “fixed” by cheating and dumping templated JSON from my staging PMM and imported them into production, and two of the panels I could fix by tweaking their JSON from ‘pmm-signlestat-panel’ to ‘singlestat’… but that’s just a quick-fix I’m not sure will survive the next upgrade, so I’m still looking for the cause of this.

dashboard.zip (10.3 KB)

Navigating to Configuration -> Plugins -> PMM -> Dashboards shows re-import options for each dashboard, but those two I imported from the staging instance don’t show.

Seems to me like the problem lies with the PMM app/plugin itself, as if it got corrupted, and consequently auto-disabled - 'cause I sure as hell didn’t disable it.


Any ideas how my data container could have lost those panels, and more importantly how to fix?

Hi, milosb

Thank you for your question and patience.

Please try next steps to solve your issue:

  1. Check if a list of dashboards is not empty docker exec pmm-server sqlite3 /var/lib/grafana/grafana.db 'select id, title from dashboard;'
  2. Go to Grafana > Configuration > Plugins > (search) PMM > Config tab > Click Disable button.
  3. Check if the of dashboards is empty docker exec pmm-server sqlite3 /var/lib/grafana/grafana.db 'select id, title from dashboard;'
  4. Stop PMM Server docker stop pmm-server
  5. Backup it docker rename pmm-server pmm-server-backup
  6. Pull latest docker pull percona/pmm-server:latest
  7. Create new container docker run -d -p 80:80 --volumes-from pmm-data --name pmm-server --restart always percona/pmm-server:latest
  8. Check if the list of dashboard is not empty again docker exec pmm-server sqlite3 /var/lib/grafana/grafana.db 'select id, title from dashboard;'
  9. For reference please follow this doc https://www.percona.com/doc/percona-monitoring-and-management/deploy/server/docker.upgrading.html#update-server-docker

Best regards

UPDATE: Belated, but better late than never. Looks like the problem was resolved after the 1.14 update.

Thanks for your help.