DevoxxUA 2021. Oleksandr Reshetnik. ETL pipeline with GCP Dataflow
The talk from Devoxx Ukraine 2021 ONLINE
Fb: www.facebook.com/DevoxxUkraine/
Website: www.devoxx.com.ua
The solution to gather data from different data sources (SQL, NoSQL, REST) related to the customer API platform. Also, data were transformed and prepared for reporting and an analytic tool like Tableau or Data Studio. The developed solution includes the reliable and flexible architecture of the data pipeline, the data model design and data dictionary for reporting, the framework for data extraction and transformation from different sources, the orchestration engine for the data pipeline, the collection of reports in Tableau.