A close look at EXPLAIN in Spark v3 - DataFriday 1x17
What happens when Spark is about to execute your transformations as you call an action? This short video will show you how to use the explain() function.
The TransformationExplainApp builds on the previous lab, #200. Chapter 4's lab \#200 was already featured in DataFriday episode 13: • How does Apache Spark optimize workloads w... . The app shows the use of the dataframe explain() method, which is the same as the EXPLAIN keyword in Spark SQL. All Spark v3 explain options are used in the code: simple, extended, codegen, cost, and formatted.
Javadoc is @ https://spark.apache.org/docs/latest/.... Spark SQL is @ https://spark.apache.org/docs/latest/....
This lab is based on chapter 4 of Spark in Action, 2nd edition (http://jgp.ai/sia).
Repository: https://github.com/jgperrin/net.jgp.b...
Lab: #500
GitHub link: https://github.com/jgperrin/net.jgp.b...
This event is supported by La French Tech Raleigh-Research Triangle ( / frenchtechraleigh .
#DataFriday