PySpark Complete Course - Learn Big Data Processing with Python
Apache Spark with Python in this comprehensive 3+ hour course. Master distributed data processing and build scalable big data applications using PySpark.
You'll Learn:
PySpark fundamentals including RDDs, actions, and transformations
Setting up Spark and running it with iPython notebooks
Working with key-value pairs and data operations (map, filter, reduce)
Performance optimization with broadcast variables and partitioning
Running Spark on clusters (Standalone, Mesos, Yarn)
Advanced features: Spark Streaming, DataFrames, SQL, and MLlib
This course covers everything from installation to advanced Spark programming. You'll start with Python and Spark setup, learn RDD operations, master transformations and actions, work with different input/output formats, and optimize performance. We also cover deploying Spark on clusters and using advanced features like streaming and machine learning.
Perfect for data engineers, data scientists, and developers working with big data. Basic Python knowledge helpful but not required.
Timestamp
00:00:00 Introduction And Course Overview
Module 1: Introduction
00:02:00 What you will learn in this Module
00:02:00 Installing Python
00:06:38 Installing iPython And Using Notebooks
Module 2: Installing Spark
00:13:06 What you will learn in this Module
00:13:06 Download And Setup
00:16:30 Running The Spark Shell
00:22:05 Running The Spark Shell With iPython
Module 3: Spark Fundamentals
00:28:43 What you will learn in this Module
00:28:43 What Is A Resilient Distributed Dataset - RDD?
00:33:37 Reading A Text File
00:37:11 Actions
00:39:24 Transformations
00:41:54 Persisting Data
Module 4: Transformations
00:46:05 What you will learn in this Module
00:46:05 Map
00:49:09 Filter
00:53:05 Flatmap
00:56:21 MapPartitions
01:00:28 MapPartitionsWithIndex
01:02:19 Sample
01:04:55 Union
01:06:06 Intersection
01:07:34 Distinct
01:09:36 Cartesian
01:12:53 Pipe
01:16:33 Coalesce
01:18:45 Repartition
01:21:14 RepartitionAndSortWithinPartitions
Module 5: Actions
01:25:12 What you will learn in this Module
01:25:12 Reduce
01:29:31 Collect
01:31:27 Count
01:34:32 First
01:35:52 Take
01:36:57 TakeSample
01:40:00 TakeOrdered
01:42:10 SaveAsTextFile
01:46:19 CountByKey
01:48:59 ForEach
Module 6: Key-Value Pair RDDs
01:52:10 What you will learn in this Module
01:52:10 GroupByKey
01:54:41 ReduceByKey
01:58:11 AggregateByKey
02:01:55 SortByKey
02:04:42 Join
02:08:58 CoGroup
Module 7: Input And Output
02:11:07 What you will learn in this Module
02:11:07 WholeTextFile
02:14:22 Pickle Files
02:18:21 HadoopInputFormat
02:23:56 HadoopOutputFormat
Module 8: Performance
02:29:27 What you will learn in this Module
02:29:27 Broadcast Variables
02:33:44 Accumulators
02:38:52 Using A Custom Accumulator
02:43:44 Partitioning
Module 9: Running On A Cluster
02:51:40 What you will learn in this Module
02:51:40 Spark Standalone Cluster
02:56:06 Mesos
02:59:44 Yarn
03:02:12 Client Versus Cluster Mode
Module 10: Advanced Spark
03:04:53 What you will learn in this Module
03:04:53 Spark Streaming
03:09:14 Dataframes And SQL
03:12:42 MLlib
Module 11: Summary & Conclusion
03:17:11 Resources And Where To Go From Here
03:18:13 Course Summary & Conclusion
-Subscribe + Hit the Bell for weekly TECH tutorials that fast-track your IT career and help you land high-paying tech jobs!
-Drop a comment below about your current networking knowledge level - We read every single one!
Show your Love & Support:
/ alphabrains
☕️ Buy us a coffee! (https://www.buymeacoffee.com/alphabrains)
Would Help me make more content like this one :)
```
If you are involved in ANY form of online collaboration, then have I got the all-in-one solution for you! Try it here (it's completely free) - https://www.taskade.com/?via=taskade
🌟*Join Our Brainers Community! *
Are you passionate about learning, tech, and courses? Join us at Discord to dive into a vibrant community where enthusiasts like you come together to share knowledge, resources, and inspiration.
🚀*Telegram*: https://t.me/+pIXV6rmfKbU5YmRk
✨*WhatsApp Channel:* https://whatsapp.Com/channel/0029V51Y...
🔥*Discord Channel:* / discord
💎*YouTube Channel:* / @alphabrainscourses