Extract and Load from External API to Lakehouse using Data Pipelines (Microsoft Fabric)
10+ hours of FREE Fabric Training: https://www.skool.com/microsoft-fabri...
Looking for Fabric consultancy? Fill in this form: https://forms.office.com/e/h5FaVBwQ7s
#datapipeline #microsoftfabric #lakehouse
In this video, I walk through how to query an external API using a Data Pipeline, extracting some JSON data and loading it into a Lakehouse in Microsoft Fabric.
This video is the first in an End-To-End Microsoft Fabric Project, you can follow the series via this Playlist: • Playlist
In my experience, this is one of THE most important and impactful things you can learn as a data engineer.
Next parts of the series include:
Transforming data in the Lakehouse using Data Engineering Notebooks.
Visualising the data in Power BI.
-LINK TO OPENWEATHERMAP-
Here's the API used in this tutorial: https://openweathermap.org/api
-OTHER VIDEOS YOU MIGHT LIKE-
LAKEHOUSE - • Microsoft Fabric Lakehouse Tutorial
Data warehouse - • Microsoft Fabric Data Warehouse Explained ...
Data pipelines - • Three data pipeline use cases to make your...
End-to-end data flows project - • Dataflows end-to-end project (Microsoft Fa...
OneLake - • OneLake - the FIRST thing you need to know...
-TIMELINE-
0:00 Intro
1:27 Exploring the API
4:10 Looking at the Lakehouse Destination
4:56 Creating the data pipeline
6:43 Source configuration
12:56 Destination configuration
15:28 Running the pipeline
16:00 Summary
6:14 Creating tables
8:09 Saving & sharing queries
10:03 Insert data
10:23 Selecting (viewing) data
10:58 Caution
11:36 Joining data
12:36 Creating views
14:42 Cross data warehouse querying
17:09 Summary
-LINKEDIN-
Not following the LinkedIn page yet? Here's the link: / learnmicrosoftfabric
-ABOUT WILL-
Hi, I'm Will! I'm hugely passionate about data and using it to create a better world. I currently work as a Consultant, focusing on Data Strategy, Data Engineering and Business Intelligence (within the Microsoft/Azure/Fabric environment). I have previously worked as a Data Scientist. I started Learn Microsoft Fabric to share my learnings on how Microsoft Fabric works and help you build your career and build meaningful things in Fabric.
-SUBSCRIBE-
Not subscribed yet? You should! There are lots of new videos in the pipeline covering all aspects of Microsoft Fabric.