I had my innodb_buffer_pool_size set to around 1 GB
This is your main problem. The buffer pool holds both read-cache and is the staging area for writes. So, if this buffer is too small, you will lots of little writes when the pool needs to make room. The buffer pool should be 80-90% of your total system RAM, but not larger than your active dataset. With 64GB of RAM, I would set this to 56GB, leaving 8GB for OS/filesystem buffers.
Next, make sure innodb_log_file_size matches 1/2 of your writes MB/hr (yes, per hr).
A search query in the crawler is returned in 10 seconds
This sounds like you are not correctly indexed. Use EXPLAIN on all of your SELECTs to determine if you see ‘ALL’ anywhere in the results. If so, this is full-table scan and you need some indexes.