HOWTO release SWAP back to OS the right way

Hi ALL, I have a master that is alerting for using up my SWAP space,
usually I add another swap file to extend the swap footprint.
or restart mysql - but since this is a Master I don’t want to disrupt the serveries
Is there another way?
I tried swapoff -a and swapon -a but it give me an error : not enough free memory to do this
I then noticed the box has 256GB RAM and only 120GB was set in innodb_buffer_pool_size
Since this variable is dynamic in 5.7
I upped it to 160GB on Wednesday and upped it again to 200GB on Thursday.
mysql> set global innodb_buffer_pool_size=164102410241024
mysql> set global innodb_buffer_pool_size=200

See pictures below
I then ran

Shouldn’t that flush out everything from the mysql memory and release the memory back?

I am now able to run swapoff -a
but it has been sitting there for hours w/o giving me back the command prompt.
Swap: 4095 3377 718
Swap: 3219 3219 0
Swap: 3209 3209 0
Swap: 2623 2623 0
Swap: 1834 1834 0

But I see the swap is being flushed out and is coming down…
I guess it will come down to Zero swap and then give back the prompt
and I can then turn on the Swap again swapon -a