This might be the same thing I was seeing for grouping, check this post.
Basically, MySQL is refusing to use in-memory temp tables for the sort because of the presence of your varchar column. You can verify if that’s the case by checking SHOW STATUS LIKE ‘Created_tmp_disk_tables’. In my grouping case, even when there were only a few rows in my tables, it would still create the temp table on disk instead of in memory.
Thing you can try:
normalizing your data (good for grouping, not so much for sorting)
changing the column to CHAR
If you can’t change the data model easily, you can try putting your server’s temp data directory on a tmpfs partition or other memory-based file system. This will speed things up (at the cost of RAM), but not nearly as much avoiding the filesort altogether.