DevFest Budapest - A Tale of Two Data Warehouses
Balázs is a Data Engineer at ingatlan.com, the market leader real estate listing site in Hungary. Previously he worked for startups and enterprises in various engineering roles and also held an advisory role at a Big Data consultancy company. He enjoys sharing experience and knowledge in meetups.
This is a talk about data warehouse migration and some of the mistakes, headaches, and nasty surprises we encountered, so you don’t have to.
We decided last year to migrate from our old on-prem data warehouse to BigQuery because we wanted to (a) simplify our data infrastructure and (b) take full advantage of other parts of our infra that were already built upon Google Cloud Platform.
Following a complete redesign and savvy tool selection (BigQuery, Stitch Data, Alooma, dbt Cloud), the loading and processing times are now orders of magnitude shorter, our operational costs have significantly decreased and our development practices have improved tremendously.
So, huge benefits to be found here; I want to highlight some of the traps and also some really cool hacks we discovered along the way.
5th November 2020
https://www.devfest.hu