Running Data Replication Pipelines on Kubernetes with Argo - #37 DoK Community
https://go.dok.community/slack
With:
Stephen Bailey - Director of Growth Analytics, Immuta
Bart Farrell - Presenter
Hundreds of data teams have migrated to the ELT pattern in recent years, leveraging SaaS tools like Stitch or FiveTran to reliably load data into their infrastructure. These SaaS offerings are outstanding and can accelerate your time to production significantly. However, many teams prefer to roll their own tools. One solution in these cases is to deploy singer.io taps and targets — Python scripts that can perform data replication between arbitrary sources and destinations. The Singer specification is the foundation for the popular Stitch SaaS, and it is also leveraged by a number of independent consultants and data projects. Singer pipelines are highly modular. You can pipe any tap to any target to build a data pipeline that fits your needs, making them a good fit for containerized workflows. This article walks through the workflow at a high level and provides some example code to get up and running with some shared templates. I also drill into reasons for choosing the Argo approach over other orchestration tools like Airflow or Dagster, and the implications from a team perspective.
About Data on Kubernetes Community (DoKC)
DoKC is an independent tech community where practitioners go to run data workloads on Kubernetes. It is managed by Constantia.io. We produce a range of resources for end users looking to run DoK including:
DoK Landscape
An interactive landscape of products and services for running data workloads on Kubernetes. (https://dok.community/landscape/)
Videos and Blogs
Hundreds of videos and blogs including how tos, tutorials, use cases, case studies, and more. (https://dok.community/landscape/) (https://dok.community/blog/)
DoK Reports
DoKC works with an independent research firm to annually produce the industry’s only DoK-focused report. (https://dok.community/dok-reports/)
Livestreams and Local Meetups
Weekly livestreams on DoK-related topics including demos, tutorials, visionary talks, panels, and more. (https://www.meetup.com/data-on-kubern...)
DoK Days @ KubeCon
We host hybrid events twice a year for people to get together and learn in-person or online. Details are published on our main website. (https://dok.community/)
Connect With Us:
https://dok.community/
/ @dokcommunity
/ dokcommunity
/ data-on-kubernetes-community
https://github.com/dokc