Beginner's Guide: Running PySpark on Windows Like a Pro (2024) | PySpark Installation Guide
Hello Everyone,
Welcome back to the channel!
Do you ever wanted to develop PySpark applications locally on your Windows machine?
In this video, I'll guide you through a step-by-step process to set up a complete PySpark development environment on Windows. PySpark is a fantastic tool for big data processing, right on your Windows machine. No need for complex clusters – I will teach you how to set up a complete local development environment to test and build your PySpark applications with ease.
By the end of this video, you'll be able to:
Install Python 3.11 and configure your environment.
Set up Java JDK 17, a crucial component for PySpark.
Download and configure Hadoop, the distributed file system for PySpark.
Install Apache Spark itself and integrate it with your system.
Finally, test your PySpark installation and run your first PySpark programs using the awesome Visual Studio Code.
Let's dive in and get started!
Setting Up The Environment:
1. Python Installation (1:00 - 2:00):
First things first, we need a strong foundation – Python! Head over to the official website, https://www.python.org/, and download the latest version of Python 3.11. Once downloaded, run the installation wizard and make sure to check the option to "Add Python to PATH." This will allow us to use Python commands directly from anywhere on our system.
2. Java JDK Installation:
Next, we need Java. PySpark relies on Java, so let's download the Java Development Kit (JDK) version 17. You can find it on the Oracle website. Once downloaded, run the installer and follow the on-screen instructions.
Here's a crucial step: After installing Java, we need to tell our system where to find it. Search for "environment variables" in your Windows search bar and open "Edit the system environment variables." Under "System variables," click "New" and create a variable named "JAVA_HOME." Set the variable value to the location where you installed Java JDK 17 (e.g., C:\Program Files\Java\jdk-17).
Don't forget to add Java to the system path as well. In the same "System variables" window, find the variable named "Path" and click "Edit." Click "New" and add the path to your Java installation directory (e.g., C:\Program Files\Java\jdk-17\bin).
3. Hadoop Installation (3:30 - 5:00):
Now comes Hadoop, the distributed file system that PySpark utilizes. There are different versions of Hadoop available. For this tutorial, we'll use a version compatible with Spark 3.3.x. You can find download options on the Apache Software Foundation website.
Once downloaded, unzip the Hadoop archive and extract its contents to a suitable location. Similar to Java, we need to set an environment variable for Hadoop. In the "System variables" window again, create a new variable named "HADOOP_HOME" and set its value to the extracted Hadoop directory (e.g., C:\Hadoop).
4. Spark Installation (5:00 - 6:30):
Finally, it's time for the star of the show – Apache Spark! Head over to the Spark website and download a version compatible with the Hadoop version you chose earlier.
Extract the downloaded Spark archive to a suitable location (e.g., C:\Spark). Now, we need to add Spark to the system path. In the "System variables" window, edit the "Path" variable and add the path to your Spark binary directory (e.g., C:\Spark\bin). Additionally, create a new environment variable named "SPARK_HOME" and set its value to the extracted Spark directory (e.g., C:\Spark).
5. Testing PySpark (6:30 - 7:00):
Let's see if everything worked! Open a command prompt
Once you will follow all the described steps, you will be good to work with PySpark locally on your computer.
Happy Learning!
@ashmanmalhotra
#pyspark #pysparktutorials #dataengineering #dataengineeringessentials