Did a 25-30 minute test and was able to actually play together, in sync, for about 30 seconds for so. The main issue wasn't latency. My latency was 10 ms and the drummers was 5ms. We were actually able to play along with each other in time, although it was brief.
The issue was the audio engine on either my side or the drummers would 'pop' then there would be about 4-5 minutes of high pitched white signal (you can see the visual disturbance in the audio signal itself- as if an audio cord is half-way plugged in) and not able to hear each other for a prolonged amount of time. Then it would reset somehow momentarily only to repeat the same process. Need to troubleshoot; this is possibly sample rate related but not sure.