Solr Cross-Datacenter Replication and Consistency at Scale: Oliver Bates, Apple Inc.
Replicating a SolrCloud index to multiple availability zones serves two primary purposes: redundancy for rapid disaster recovery and data locality for minimizing request latencies across different regions. These features are particularly interesting for services where SolrCloud is used as the primary data store or when availability of the search index is directly linked to uptime. In our view, a reliable cross-availability zone replication system possesses two qualities. First, it should be fault-tolerant and provide certain guarantees about data loss and availability, even in the event of a complete datacenter outage. Second, it should be capable of detecting and resolving any eventual inconsistencies. To the first of these ends, we developed a Solr plugin which uses a distributed queue to achieve non-blocking, failure-tolerant writes without compromising local indexing performance. To the second, we are currently working on a time-based Merkle tree comparison technique to detect and resolve inconsistencies during online indexing. In this talk, we will present the design of these components as well as the overall system architecture, and discuss their guarantees and limitations in the context of similar efforts in the community.
Presented at Lucene/Solr Revolution. Learn more: https://activate-conf.com/