PySpark Learning Series | 12- Some useful FILE SOURCE OPTIONS to remember
PySpark Learning Series | 12- Some useful FILE SOURCE OPTIONS to remember
In this video we will learn some useful file source options that can be used to mitigate errors while reading files and creating dataframes.
These file options are ignoreCorruptFiles, ignoreMissingFiles, pathGlobFilter, recursiveFileLookup and few CSV options