Data pipeline tutorial 5 - Automation with Starburst and dbt | Starburst Academy
Learn more about data pipelines: https://academy.starburst.io/explorin....
This video is part 5 of a 5-part series showing how data pipelines using modern data lakes are constructed using Starburst Galaxy and SQL. To do this, we will continue using the BlueBikes dataset: https://www.bluebikes.com/ and add dbt Cloud to automate the processes created in the Land, Structure, and Consume layers in the previous videos. In a data lakehouse, automation is often desirable in a data pipeline fits with the three-part modern data lake structure observed in the previous tutorials. Using this approach, data engineering tasks can be made more streamlined using the Starburst Galaxy-dbt Cloud integration. This allows the data inside the pipeline to reach its final destination faster for use in queries, or as the basis for BI tools and dashboards. With Starburst Galaxy, integration with dbt Cloud is easy, and this step-by-step video tutorial shows you how to get started.
Starburst Galaxy makes creating data pipelines much easier than many platforms because it allows you to use SQL to perform many tasks that might normally require a general scripting language like Python. As you progress in this video training series, you can follow along with your own Starburst Galaxy account: https://www.starburst.io/platform/sta.... The aim is to provide a big data tutorial that lets you test out the software on real world datasets.
For more information on this topic check out the full course, Exploring data pipelines: https://academy.starburst.io/explorin.... It digs deeper into data pipelines at all levels, including a discussion of how Starburst Galaxy can help construct a pipeline using SQL. It is available on Starburst Academy: https://academy.starburst.io/.
The course is part of the Data foundations series: https://academy.starburst.io/page/dat.... The series provides a conceptual and contextual background to help you understand how Starburst fits within Big Data ecosystems, including Exploring data lakes: https://academy.starburst.io/explorin... and Exploring data lakehouses: https://academy.starburst.io/explorin.... Each course in the series is suitable for all audiences.
#datapipeline #etl #etlpipeline #datalake #datalakehouse #trino #starburst #dataengineering #datalake #sql #sqldatapipeline #data #datascience #dbt #dataautomation #datagovernance