When MySQL grabs memory, it tends to not give it back. That does not mean MySQL is actively “using” all of the memory; it is simply holding onto the memory so that it can use the memory later if needed. If MySQL is the only application running on the server, then you should be okay as MySQL will manage the memory and do with it as it sees fit. However if MySQL ends up using even more memory that you have free, then you could run into problems, so you need to see what your “high water mark” is and lower memory usage as needed to prevent the system from being overran.
If you are running another memory hungry application on the same server, then you could get into trouble. If that is the case, then you should ideally put MySQL on it’s own server, otherwise you’ll need to lower the memory footprint of MySQL (most likely decrease the InnoDB buffer pool size) to prevent memory contention between applications.
As far as innodb_max_dirty_pages_pct, I do not believe that will lower the currently used memory footprint of MySQL. All that does is write changes from the InnoDB buffer pool to disk. Setting this variable to zero is typically only done when you are planning to shutdown the server and want there to be fewer changes to write to disk once you actually perform the shutdown so that it happens quicker. This is “safe” to do in production, as long as you understand the performance implications of it since you are writing more to disk at that point. If your system is not very busy, and you have free I/O time, then it likely will not cause an issue. However if you run a busy system, you could notice a performance impact when setting that variable to zero.
So to specifically answer your question, I do not know of any way to release a significant amount of memory from MySQL without restarting it. The only mostly insignificant thing I’ve seen is resetting the query cache, but most people do not even use the query cache these days.