I thought I would try something with the replica that showed behaviors of having issues and chained another replica to it. The backup from 1st tier replica did not include GTID information (using innobackup) even though GTID is turned on and so I ran the change master command with CHANGE MASTER TO MASTER_AUTO_POSITION=1; with the binlog name in the backup but left the position out. Started the Slave and to my surprise the slave thread connected and show slave status shows Slave_IO_Running: yes and Slave_SQL_Running: yes, no errors and the Master_Log_File is correct and the Read_Master_Log_Position keeps up with the 1st level replica. Looked completely clean to me except I expected it to have Retrieved_Gtid_Set and Executed_Gtid_Set to have values. The unexpected behaviour was this backup was from 3 days ago on a busy servers so I expected seconds behind to be in the days to catch up but it connected with 0 seconds behind.
Expecting something is still wrong I decided to count lines in a busy table and found that the table was missing 40K rows. I ran the server for another day and the 40K row stayed consistent and the show slave status showed that the database was still following the master log position. I was able to connect a replica to a another server without error but in fact I had lost a lot of data…not cool.