Martina Ivanicova: How Data Mesh Drives Analytics in Kiwi | Data Science Meetup Bielefeld
Some say 80% of data scientist’s time is spent on looking for data and preparing data. To have well documented, curated data at tip of our fingers is a dream of many of us. The roadblock we face in achieving it is the fact that with micro-service architecture the operational data are split among domains, while with analytical data the situation is different as the traditional mindset to keep them in one central place, one piece of infrastructure is prevalent. The friction between these two concepts is what we had observed also in Kiwi:
There was no contract between producers and consumers of the data, which was often manifested as “bad data quality”. What it actually meant is that consumers’ expectations differ from reality. Because they are accessing raw, undocumented, non-curated data, with not announced schema changes and content changes ...
One central team was managing ETL pipelines from all the data sources to one central Datalake. The more data sources, the bigger central team we needed for the maintenance
This talk walks you through how Kiwi decided to adopt some of the Data mesh concepts to address the dichotomy between operational and analytical data by introducing topology based on domains not stack.
Martina has 12+ years industry experience spanning from engineering traditional data warehouses, through architecting of cloud native IoT data solutions to data platform management. Her passion lies in removing silos - in both technology and social sense