The Unified Theory of Batch and Stream Processing
I met Santona Tuli at Data Day Texas 23 where she presented the unified theory of batch/stream processing (spoiler: It is all just SQL). Naturally, I had to invite her to the podcast to discuss in detail.
We discussed the reasons behind batch/stream unification, how to determine if a process should be batch or stream, the rise of the real-time applications and Santona's advice on building them reliably.
Last but not least, we discussed the challenges around building a data-processing SaaS, especially as customers demand to maintain their full ownership of all the data.
00:00 Intro
01:17 Why unify batch and stream processing?
03:11 It is all about data sources
06:05 Data Products
12:55 The case for real time website analytics
14:44 The real-time attribution problem
23:00 Pipeline Synchronization
26:56 Event time or Wall-Clock Time?
30:00 Experiments with Streams and S3
34:00 Running Data Processing as a Service?