How to create a low-cost Spark cluster on Azure
In this video, I demonstrate how to provision a low-cost Apache Spark cluster on the Microsoft Azure platform by using the Azure Distributed Data Engineering Toolkit (AZTK).
The Azure Distributed Data Engineering Toolkit (AZTK) is a python CLI application for provisioning on-demand Spark on Docker clusters in Azure. It's a cheap and easy way to get up and running with a Spark cluster, and a great tool for Spark users who want to experiment and start testing at scale.
The toolkit is built on top of Azure Batch but does not require any Azure Batch knowledge to use.
Notable Features of AZTK
Spark cluster provision time of 5 minutes on average
Spark clusters run in Docker containers
Run Spark on a GPU enabled cluster
Users can bring their own Docker image
Ability to use low-priority VMs for an 80% discount
Mixed Mode clusters that use both low-priority and dedicated VMs
Built in support for Azure Blob Storage and Azure Data Lake connection
Tailored pythonic experience with PySpark, Jupyter, and Anaconda
Tailored R experience with SparklyR, RStudio-Server, and Tidyverse
Ability to run spark submit directly from your local machine's CLI
The command I used in the Azure Cloud Shell to setup the required Azure resources is:
wget -q https://raw.githubusercontent.com/Azu... -O account_setup.sh && chmod 755 account_setup.sh && /bin/bash account_setup.sh