Common Transformations in PySpark & SQL | Select, Filter, Distinct, Union, Repartition vs Coalesce
In this video, we’ll explore some of the most commonly used PySpark transformations and understand them in both PySpark (code) and SQL (query) formats. This dual approach helps you master Spark from both the coding and analytical perspectives.
All code and data files are available on the below path:
https://github.com/databeli/pyspark_c...
PowerPoint Presentaion useed in the complete playlist(27 slides)
https://topmate.io/narender_kumar_91/...
What you’ll learn:
How to read tables in PySpark and SQL
Working with columns — select, add, delete, rename
Using expressions (expr, alias, literal) for column operations
Changing data types with cast()
Filtering data with where() and filter()
Removing duplicates using distinct()
Combining data using union and unionAll
Sorting results with orderBy()
Limiting results with limit()
Understanding repartition vs coalesce — when to use which and how they impact shuffling
How Spark handles partitions and parallelism
Using collect() to bring data to the driver
Converting PySpark DataFrame to SQL (createOrReplaceTempView)
Converting SQL query to DataFrame (spark.sql())
Best practices for when to use PySpark vs SQL for performance and flexibility
By the end of this video, you’ll know how to perform essential data transformations in Spark, optimize performance using partitioning concepts, and seamlessly switch between SQL and PySpark code.
#pyspark #pysparktutorial #databricks #databrickstutorial #sparktransformations