du it in Rust: async, tokio, streams, and surprises about perf
In this video I implement a toy clone of `du` utility in Rust in 3 different ways:
using standard library and blocking APIs,
using async and tokio, but naive sequential processing,
using more advances async features (streams, FuturesUnordered, select) to extract more parallelism out of the async code.
Finally, all versions are benchmarked on Windows, WSL2, and Linux with surprising performance results!
The code and benchmarks are on Github: https://github.com/artempyanykh/rdu
Contents:
00:00:00 - Intro
00:00:53 - Plan
00:01:53 - Start coding
00:02:17 - Using clap for CLI
00:08:20 - Scaffolding for the app
00:14:37 - Blocking/sync implementation of `calc_space_usage`
00:17:53 - Rant about iterators, docs, and too specific types
00:20:04 - Back to code
00:22:46 - Adding `humansize`
00:24:36 - Debugging differences between du and rdu
00:29:16 - Benchmarking with hyperfine
00:31:16 - Benchmark results: du vs rdu-sync
00:31:57 - Setting up a multi-binary crate
00:35:29 - Naive async version: rdu-async-seq
00:43:52 - More sophisticated async version: rdu-async-par
00:58:48 - Streams and select!
01:09:11 - rdu-async-par full impl recap
01:12:15 - Benchmarks: full
01:16:38 - Summary: async as the default
01:19:05 - Summary: async Rust ergonomics
01:20:41 - Summary: performance