ETL Processing on Google Cloud Using Dataflow and BigQuery Python
Overview
Dataflow is a Google Cloud service that provides unified stream and batch data processing at scale. It is built on the Apache Beam project, which is an open source model for defining both batch and streaming data-parallel processing pipelines. Using one of the open source Apache Beam SDKs, you can build a program that defines the pipeline and then use Dataflow to execute the pipeline.
In this lab, you use the Apache Beam SDK for Python to build and run a pipeline in Dataflow to ingest data from Cloud Storage to BigQuery, and then transform and enrich the data in BigQuery.
Code: https://github.com/starttraining/ETL-...