Implementing data center to data center replication for a distributed database by Ewout Prangsma
ArangoDB is a scalable, distributed multi-model database. However, for this talk, it is not necessary to know what this means. Rather the only crucial fact is that it is distributed and written in C++.
Before you stop reading: This talk is about a golang success story. Namely, we had to implement resilient data center to data center (DC2DC) replication for ArangoDB clusters from scratch within 6 weeks (plus some time for testing and debugging). Therefore, we built upon – ArangoDB’s HTTP-based API for asynchronous replication, – the existing golang driver, – the fault tolerant scalable message queue system Kafka, – a lot of existing golang libraries and – golang’s fantastic capabilities for parallelism, communication and data manipulation and pulled this task off. This talk is the story of this project with its many challenges and successes and ends with a surprising revelation about which of the above we did not actually need in the end.