Automating Ambiguity Managing dynamic source data using dbt macros
We've been huge fans of dbt since overhauling our data stack 2 years ago. As product development cycles have increased in velocity, though, I began to notice that changing data schemas were causing huge amounts of work in order to update downstream reporting for execs and other teams. Behavioral data is never static, but it is non-trivial to build a system that requires little or no maintenance for data engineers and analysts.
In this talk, I'll tell the story about our stack, our changing schemas and how I used dbt to make life easier for both the engineers instrumenting tracking as well as the executives consuming downstream reporting.
Catch all of Coalesce 2021, the analytics engineering conference, at https://getdbt.com/coalesce-2021/.
00:00 Introduction
05:16 Problem: Data is messy
08:36 Our tech stack
10:33 Solution: One dbt macro to rule them all
13:11 Union macro logic
15:39 What's the big deal?
22:15 Current limitations and future additions