Spark Installation | PySpark Installation | Windows 10 / 11 | Step by Step |

Опубликовано: 20 Октябрь 2024
на канале: DEwithDhairy

2,594

In this video we will be setting up #python #java #spark #pycharm and #pyspark in our local system.

Steps :
======

1) #Java Download:
https://www.oracle.com/java/technolog...

2) #Python Download: (3.11.4) :
https://www.python.org/getit/windows/

3) #Spark Download: (3.4.2) :
https://www.apache.org/dyn/closer.lua...

4) winutil.exe (3.3.5 version) :
https://github.com/cdarlint/winutils

5) #Pycharm community download:
https://www.jetbrains.com/pycharm/dow...

Check Python , Java , PySpark and Spark Version :
========================================
python --version
java --version
spark-shell
pyspark --version

Error : PYCHARM Error-- java.io.IOException: Cannot run program "python3": CreateProcess error=2, The system cannot find the file specified.

Solution :
========
Write these below two lines before the spark object creation.

1) os.environ['PYSPARK_PYTHON'] = sys.executable
2) os.environ['PYSPARK_DRIVER_PYTHON'] = sys.executable
-------------------------------------------------------------------------------------------------------------------------------------------------------

If we don't want to use the virtual environment python then.
add the below environment variable.

Variable Name : PYSPARK_PYTHON
Variable Value : C:\Users\{your_user_name}\AppData\Local\Programs\Python\{PYOUR_PYTHON_VERSION}\python.exe

if you add the PYSPARK_PYTHON variable then you will not required to set the OS environ variables in the code.
-----------------------------------------------------------------------------------------------------------------------------------------------------
Sample Code :
============
from pyspark.sql import SparkSession
from datetime import datetime, date
from pyspark.sql import Row
import os
import sys

os.environ['PYSPARK_PYTHON'] = sys.executable
os.environ['PYSPARK_DRIVER_PYTHON'] = sys.executable

print(sys.executable)

spark = SparkSession.builder.getOrCreate()

df = spark.createDataFrame([
Row(a=1, b=2., c='string1', d=date(2000, 1, 1), e=datetime(2000, 1, 1, 12, 0)),
Row(a=2, b=3., c='string2', d=date(2000, 2, 1), e=datetime(2000, 1, 2, 12, 0)),
Row(a=4, b=5., c='string3', d=date(2000, 3, 1), e=datetime(2000, 1, 3, 12, 0))
])

df.show()

Let's connect on LinkedIn :   / dhirajgupta141

DSA In Python Interview Series :    • dsa for data engineer | dsa for data ...

PySpark Interview Series :    • pyspark interview questions and answers

Pandas Intervew Series :    • pandas interview questions and answers

SQL Interview Series :    • sql interview questions and answers

#python #leetcode #dsa #interview #sql #dataengineers #dataanalytics #datascience #StrataScratch #Facebook #data #dataengineeringinterview #codechallenge #datascientist #pyspark #CodingInterview
#dsafordataguy