Beam Summit 2023 | CI/CD for Dataflow with Flex Templates and Cloud Build - Mazlum Tosun
The goal of this talk is to show a full example of a CI CD pipeline for Dataflow jobs.
The jobs will be based on Flex Template which is a way to standardize the deployment of Dataflow jobs and we are going to show a full example with Java and Python SDKs.
The CI CD will be orchestrated with Cloud Build based on a GitHub repository:
Launch unit tests on push to the Github repository
Manual job to deploy the Dataflow job with Flex Template
Manual job to run the Dataflow job and template
In an extra and optional part, we will show the Dataflow Flex Template deployment with Dagger IO and Go.
Dagger is a tool that allows to write CI CD Pipeline As Code.