12 Understand Spark UI, Read CSV Files and Read Modes | Spark InferSchema Option | Drop Malformed
Video explains - How to read CSV Files? How Spark works in background while reading files? Understand Spark UI. What is InferSchema? What are the read modes available in Spark?
Chapters
00:00 - Introduction
01:29 - How to read CSV file in Spark
02:29 - What is happening in Spark UI ?
03:15 - Read header from file
04:00 - Spark InferSchema Option
06:02 - Read file with Schema
08:27 - CSV File read options
08:54 - Read Modes
09:25 - Permissive Mode
13:14 - Drop Malformed Mode
14:23 - Fail Fast Mode
15:35 - Use case for Read Modes
16:05 - Bonus Tip
Spark CSV Documentation - https://spark.apache.org/docs/latest/...
Local PySpark Jupyter Lab setup - • 03 Data Lakehouse | Data Warehousing ...
Python Basics - https://www.learnpython.org/
GitHub URL for code - https://github.com/subhamkharwal/pysp...
The series provides a step-by-step guide to learning PySpark, a popular open-source distributed computing framework that is used for big data processing.
New video in every 3 days ❤️
#spark #pyspark #python #dataengineering