Pushdown filter on CSV ingestion with Spark v3 - DataFriday 1x15
Spark v3 has added a pushdown filter option when ingesting a CSV file.
Pushing down a filter allows to have the execution plan run a smarter ingestion. In this video, I am going very quickly over the normal ingestion, then looking at how you can add a filter. Finally, the last lab measures performance between Spark v2.4.6 and v3.0.0 when you ingest a larger CSV file. I will quickly go over chapter 7's lab #200 and look at lab #201 and lab #991, which are new additions (i.e. not in the book).
These labs are inspired from chapter 7 of Spark in Action, 2nd edition (http://jgp.ai/sia).
Repository: https://github.com/jgperrin/net.jgp.b...
Lab: #200
GitHub link: https://github.com/jgperrin/net.jgp.b...
Lab: #201
GitHub link: https://github.com/jgperrin/net.jgp.b...
Lab: #991
GitHub link: https://github.com/jgperrin/net.jgp.b...
This event is supported by La French Tech Raleigh-Research Triangle ( / frenchtechraleigh .
#DataFriday