03 Spark Streaming Local Environment Setup - Docker, Jupyter, PySpark and Kafka
Video covers - How to setup Spark Streaming environment in Local Machine using Docker?
Chapters:
00:00 - Introduction
00:40 - Docker Desktop Installation
01:14 - Github clone
01:43 - Docker Setup
03:02 - PySpark Jupyter Lab Setup
URLs:
Docker Desktop: https://www.docker.com/products/docke...
Github URL: https://github.com/subhamkharwal/dock...
Jupyter Lab URL: http://localhost:8888/
Spark UI URL: http://localhost:4040/
Kafka bootstrap server URL: ed-kafka:9092
Docker command:
docker compose up
Kafka command:
To view all topics: kafka-topics --list --bootstrap-server ed-kafka:9092
To create a new topic: kafka-topics --create --topic test-topic --bootstrap-server ed-kafka:9092
Keywords: Apache Spark, PySpark, Spark Streaming, Real-time Data Processing, Data Streaming, Big Data Analytics, PySpark Tutorial, Apache Spark Tutorial, Streaming Analytics, Spark Structured Streaming, PySpark Streaming, Big Data Processing.
New video in every 3 days ❤️
Make sure to like and Subscribe.