Setup Jupyter Notebook with Pyspark
Pre-requisites:
1. Download Docker -- https://docs.docker.com/desktop/
2. Download Python(3.12.10) -- https://www.python.org/downloads/(Make sure you select to add python.exe to Path option while installing python)
Setup:
Once all requirements are installed, Pull the repository:
git clone https://github.com/LearnBe4/spark-internal...
Go to root directory of project and run the following command to install the dependencies and bring up the docker container:
uv sync
docker compose up # This will take sometime to build the image
Web UIs
The Jupyter Notebook: `localhost:8080`
Token: jupyter
Spark UI: `localhost:4040`
Spark Master Node: `localhost:9090`
Spark History Server: `localhost:18080`