Make a bigger pie: Updating the partition count in a consistent hash ring
John Dickinson
http://lca2018.linux.org.au/schedule/...
Consistent hash rings are often used in distributed storage systems to efficiently compute data placement. However, for efficiency, these hash rings are often implemented with a fixed number of partitions of the key space. This is how OpenStack Swift's data placement algorithm works. Swift's "part power" is used to segment, or partition, the consistent hash ring used for placement. The part power is set when the ring is created, and it determines the upper limits of capacity in the cluster.
Unfortunately, if the partition count is set too high, there will be excessive overhead in the system and storage capacity will be underutilized. However, if it is set too low, then data placement will get "lumpy" and be unevenly distributed across the cluster. An operator is forced to balance these two concerns, often without knowing how much the cluster will grow.
Recently the Swift community has merged a feature that allows operators to increase the partition power in an existing cluster. This allows operators to set the part power to a low value when the cluster is created and increase it over time as the cluster expands. Implementing this feature is tricky, because the part power is a fundamental part of the data placement algorithm. Changing it can potentially make all of the data in the cluster completely inaccessible. Swift's implementation, however, is able to keep the cluster available during the entire process.
This talk will describe the data placement problem, explain how the community implemented the feature, and give a short demo of the feature in action.
This talk was given at Linux.conf.au 2018 (LCA2018) which was held on 22-26 January 2018 in Sydney Australia.
linux.conf.au is a conference about the Linux operating system, and all aspects of the thriving ecosystem of Free and Open Source Software that has grown up around it. Run since 1999, in a different Australian or New Zealand city each year, by a team of local volunteers, LCA invites more than 500 people to learn from the people who shape the future of Open Source. For more information on the conference see https://linux.conf.au/
#linux.conf.au #linux #foss #opensource