We currently have a mysql master/slave replication set where the slave has gone into a lag of about 250 000 seconds.
We are running mysql 5.7.14 (from percona repo) and gtid and parallel master and slave processing is enabled.
During the weekend a LOT of data was processed and generated about 300Gb worth of binary logs that has to be processed, which is all good and well.
The two servers are in the same network subnet , connected via gigabit ports on the same switch.
I noticed that the slave_io_thread seems to be lagging as it is very far behind compared to the master server binary logs.
As a test, I disabled the slave sql_thread and let it only sync binary to relay logs.
Looking at the network throughput , we are only getting 6Mb/s
After it synced about 25Gb of data which took an entire night, I enabled sql_thread and within an hour it was all processed.
I initially though that this was a networking issue, however, when doing an scp from master to slave server writing to the same disk where the relay logs are located, then I get a throughput of about 75-80Mb/s
I have tried to enable slave_compressed_protocol however this made it even worse.
I can clearly see that the io_thread is the bottleneck, but I am out of ideas on how to speed this up.
Any ideas here would be greatly appreciated.