What is Data Lake? Tutorial with AWS S3, Glue & Athena
Let’s build a data lake with AWS S3, Glue & Athena! Aggregate data from different sources into one place and make it queryable through S3!
Our original data sources are going to be CSV files, DynamoDB tables and relational tables.
We will use S3 as the centralized store for the data lake. We will use Glue to discover data schemas automatically. Finally, we will use Athena to query all our data using SQL.
Data Lake makes data from all across your organization/app available in one central hub and makes it queryable using SQL so that you can run complex queries on top of all available data.
0:00 Architecture
01:50 Ingest CSV into S3
03:05 Discover CSV schema using Glue Crawlers
06:40 Discovered Tables
07:05 Query CSV using Athena in SQL
08:25 DynamoDB Tables
11:12 Export DynamoDB tables into S3
15:20 Glue Crawler for DynamoDB data
18:12 Query DynamoDB data using Athena in SQL
21:47 Design Summary
#dataLake #aws #systemDesign
Visit me at: https://irtizahafiz.com
Reach me at: [email protected]