1 BILLION row challenge in Go - 2.5 Seconds!
In this video we look at how we can aggregate 1 billion rows of weather station data in as little time as possible. We start with a naive approach and optimize to go from 1m30 seconds to 2.5 seconds. Using the power of memory mapped files, a custom hash map implementation, and multiple Goroutines.
Implementation: https://github.com/duanebester/1brc-go
Thanks to Ben Hoyt
https://benhoyt.com/writings/go-1brc/
00:00 Intro
00:54 Simple Implementation
09:55 Advanced - Using mmap
15:01 Custom integer parsing
22:12 Parallel processing
34:16 Custom hashmap
42:48 Results