PMM Upgrade fails when switching grafana from sqlite to postgres

Percona Monitoring and Management (PMM) PMM 2.x
1

Hey,

I just wanted to upgrade our (dockerized) PMM instance from version 2.35 to the latest version.

This includes (according to the changelog) the migration from sqlite to postgres for the grafana datastore.
The changelog: PMM 2.37.0 - Percona Monitoring and Management does not state any manual steps required when performing the upgrade.

After the upgrade, grafana refused to start and logged the following error:

logger=sqlstore t=2023-06-07T11:12:19.583969017Z level=info msg=“Connecting to DB” dbtype=postgres
logger=migrator t=2023-06-07T11:12:19.589114973Z level=error msg=“alert migration failure: could not get migration log” error=“failed to check table existence: pq: role "grafana" does not exist”

I checked the postgres instance shipped with PMM and can confirm: There is neither a grafana user, nor a grafana schema.

How can/shall we migrate our grafana sqlite-file towards postgres?

EDIT: I found a comment in JIRA about manually migrating dashboards. I used pgloader to copy the data over from sqlite to postgres. Grafana was not happy about that:

logger=plugindashboards t=2023-06-07T12:53:27.849554117Z level=info msg=“Syncing plugin dashboards to DB” pluginId=pmm-app
logger=plugindashboards t=2023-06-07T12:53:27.850678326Z level=error msg=“Failed to load app dashboards” error=“pq: operator does not exist: bigint = boolean”
logger=infra.usagestats.collector t=2023-06-07T12:53:27.850709466Z level=info msg=“registering usage stat providers” usageStatsProvidersLen=2
logger=provisioning t=2023-06-07T12:53:27.869259165Z level=error msg=“Failed to provision data sources” error=“Datasource provisioning error: pq: invalid input syntax for type bigint: "false"”
Failed to start grafana. error: Datasource provisioning error: pq: invalid input syntax for type bigint: “false”
Datasource provisioning error: pq: invalid input syntax for type bigint: “false”

Could maybe be because I was not able to edit the mentioned “import-dashboards.py” file, because I have no idea, where this file lives.

Thanks a lot

1 Like
2

Did you do a standard “docker pull” of the latest imago or did you use the upgrade button in the UI to go from 2.35.0 to 2.37.0? One of our engineering leads was baffled by this error as the code to make the move was removed from the released version (the release note fell through the crack somehow) and is planned to go out in 2.38.0 instead. Clearly something was missed that got you into this state so now we’re trying to figure out what.

3

Hey Steve,

I did a standard “docker pull”, and did not use the uprade-Button in the UI.

What I maybe did not make clear:
As we use LDAP for grafana-Logins, we need to maintain our own grafana.ini.

During my first upgrade-attempt, I did not apply any changes to the grafana.ini file.
But that caused errors, as the grafana sqlite DB was constantly “locked”, and therefore grafana refused to start / restart

I then tried copying over this bit from git: pmm/grafana.ini at main · percona/pmm · GitHub

If the migration from sqlite to postgres is actually not included in that release, it would totally explain, why the PG-DB & User was not automatically created :smiley:

My main motivation for that upgrade is/was that we experience big issues with PMM consuming too many resources (actually CPU).
We monitor hundreds of MySQL instances with it, and currently face a server load on pmm exceeding 1000, while the machine has only 16 CPU cores.
That causes grafana to crash / behave strange and logging us out constantly. Our hop was/is, that once grafana does not rely on sqlite any more, this improves at least a bit.