AWS Services used for Data Engineering | AWS Services that every Data Engineer must know
In this video , i discuss the important AWS Services which are used in Data Engineering projects.
The following services are discussed :
1. Data Ingestion / Streaming
AWS Kinesis Data Streams
AWS Kinesis Firehose
AWS Database Migration Service
AWS MSK ( Manages Service for Kafka)
2. Data Processing / ETL
AWS Glue
AWS Lambda
AWS EMR (Elastic MapReduce)
AWS Batch
3. Data Storage / Database
AWS S3
AWS RDS
AWS DynamoDB
AWS AuroraDB
4. Data Warehouse / Analytics
AWS Redshift
Athena
AWS Redshift Spectrum
5. Orchestration and Workflows
AWS Step functions
AWS MWAA (Managed workflows for Apache Airflow)
AWS Data Pipeline
6. Version Control and CI/CD
AWS CodeCommit
AWS CodePipeline
#aws #dataengineering #dataengineer