PySpark Learning Series | 02-Spark Installation on Windows
PySpark Learning Series | 02-Spark Installation on Windows
In this video we will learn how to install Spark on windows machine and see how to run a simple spark program using spark shell.
We will see the complete PySpark setup along with compatible Java, Python and Winutils versions
Install Python Video: • How to Install Python and Set Environment ...
Install Java8 or later: https://www.java.com/download/ie_manu...
Install Apache Spark 3.3.0: https://archive.apache.org/dist/spark/
Install Python 3.10.9: https://www.python.org/downloads/rele...
Download winutils and hadoop.dll: https://github.com/kontext-tech/winut...
Set Environment Variables:
HADOOP_HOME = C:\hadoop
JAVA_HOME = C:\java\jdk
SPARK_HOME = C:\spark\spark-3.3.0-bin-hadoop3
PYSPARK_PYTHON = C:\Users\4uasa\AppData\Local\Programs\Python\Python310\python.exe
Set Path:
%JAVA_HOME%\bin
%HADOOP_HOME%\bin
%SPARK_HOME%\bin
Sample Pyspark Code:
import pyspark
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName('test').getOrCreate()
columns = ["language","users_count"]
data = [("Java", "20000"), ("Python", "100000"), ("Scala", "3000")]
df = spark.createDataFrame(data=data, schema=columns)
print(df.printSchema())