On databases where most of the allocated RAM is always used it is very bad to let the DB process get so big that the OS starts to use the swap.
And the general rule is that it’s better to use a bit less memory than risking that the OS starts to swap.
So 2.8GB allocated out of 3G sounds very good.
When your server starts to go slow does it have high CPU or high I/O load?