Run Apache Spark Jobs on Kubernetes with Spark Operator and Airflow 3.0
Learn how to run Apache Spark applications on Kubernetes using the Spark Operator and orchestrate them with Apache Airflow 3.0. This step-by-step tutorial covers installing the Spark Operator with Helm, submitting Spark jobs declaratively using YAML, organizing your Airflow Git repository, configuring RBAC, and using the SparkKubernetesOperator inside a DAG.
By the end of this video, you’ll be able to deploy and manage Spark workloads in a Kubernetes-native way and automate them using Airflow—ideal for data engineers, DevOps teams, and platform engineers building scalable data pipelines.
🔗 Related Links:
• Airflow 3 on Kubernetes Guide: https://nsalexamy.github.io/service-f...
• GitHub Repo: https://github.com/kubeflow/spark-ope...
• Sample DAG and Spark YAML examples included
📧 For questions, reach out to Young Gyu Kim at [email protected].