@steve.hoffman I created a Jira bug for the above issue. Could you please follow up with the dev team and fix it in the 2.39 version as we are unable to use the PMM because the UI is not responding? Otherwise, we have to remove the nodes from the PMM server.
NOTE: I had removed a few nodes, and now Load has been reduced and the UI is responding fine. Now, I am running with 60 Nodes for testing purposes, and PMM is doing well. Again, I added around 40 servers, then again, same issue.
I can only promise the team will review and prioritize the issue accordingly. If we discover a workaround or config change as a fix we will get that out right away but I can’t promise a fix in any particular release version. If you are a Percona customer please escalate through your CSM and we possibly can issue a custom build until it’s fixed in a public release.
Unlike the other journaling modes, PRAGMA journal_mode=WAL is persistent. If a process sets WAL mode, then closes and reopens the database, the database will come back in WAL mode. In contrast, if a process sets (for example) PRAGMA journal_mode=TRUNCATE and then closes and reopens the database will come back up in the default rollback mode of DELETE rather than the previous TRUNCATE setting.
The persistence of WAL mode means that applications can be converted to using SQLite in WAL mode without making any changes to the application itself. One has merely to run "PRAGMA journal_mode=WAL;" on the database file(s) using the command-line shell or other utility, then restart the application.
The WAL journal mode will be set on all connections to the same database file if it is set on any one connection.
I am thinking and was going down the path of looking to see that maybe it wasnt grafana but something in containerd (While I am also VERY new to) I looked at the containerd config file and said yep. someone other than me needs to figure out what each of ones do… . I have not eyet looked to see if sqllite is configured in WAL mode or not…
It looks like the default is DISABLED and at least a quick glance at SQLite Docs here https://sqlite.org/wal.html
Seems like maybe we should move it to True when more than XXX nodes. I am willing to try this
It looks like that is a setting in the grafana.ini file that might want to be set initially… But we both said the same thing
@Naresh9999 The only thing (me just being me) is what changed I am “guessing” you are on one of the newer versions… If you had 500 before and no issues why see it now with 100?
Hi @Naresh9999 can I ask 2 questions?
Have you migrated to service accounts? Is it possible that one of your old PMM Clients are still configured to old server or have incorrect API key and keeps trying to connect to PMM Server?
@nurlan I have never migrated to service accounts. This is a new server I have created from scratch.
Is there any way that I can identify which server is trying?
for What it is worth… I did a search for grafana.db inside the docker and found two of them… Not quite sure why or if somehow I did it…
But Once I did pragma journal_mode=wal on both files I have not seen it (yet) I did NOT think to look at the other potential settings prior to me setting this
Additionally I did find that when I work with the Alerting aspect quickly I tend to see it happen more often (modifying alerts after using a Template base that comes with pmm)
While I do NOT know if it is suggested I have not seen the issue since I did the following as well as the above (Again on each grafana.db file as I had two but was not sure WHY I had two)
Do this at your own risk…**** I only did the first three .
When you issue the commands you MUST be in the directory that the grafana.db file is in… Else it does nothing
sqlite> pragma journal_mode=wal;
sqlite> PRAGMA cache_size = 10000; -- Set cache size to 10000 pages
sqlite> PRAGMA synchronous = OFF; -- Disable synchronous write (less durable, more performance)
sqlite> PRAGMA locking_mode = EXCLUSIVE; -- Use exclusive locking mode
sqlite> PRAGMA busy_timeout = 1000; -- Set a timeout of 1 second for waiting on locks
sqlite> PRAGMA temp_store = MEMORY; -- Use memory for temporary storage (if feasible)
Also if you are like me and new to Docker Make sure you commit the changes as well (and then I normally restart the docker)