I have such a problem: one large MEMORY table and mysqld with memlock enabled (“show status” shows “locked_in_memory ON”). When I start mysqld everything seems like OK: “select …” from the table is very fast and mysqld process SWAP size is 9-10 mb. And at this moment I have about 200-300 mb of free memory (I use top to find it out). But in some time mysqld SWAP size is increasing up to 300-500 mb, and queries to the MEMORY table get very slow. Stopping other services like httpd on this server can help sometimes, but not very much, anyway, mysqld SWAP size is increasing and the performance is awful.
The OS on the server is RHEL 4. Mysql version is non rpm 5.1.15 (I also tried 5.1.14 and 5.0.27).