Spark performance: measuring cache and checkpoint with a Mac and a KFocus - DataFriday 1x22
Performance in Apache Spark is always a complex topic, in this short video, I investigate the various solutions I can use: cache and checkpoints and compare the executions on two laptops.
I'll first go quickly through the directed acyclic graph (DAG) with transformations and actions, then, after describing where the optimization can happen in Apache Spark, I am executing the lab on two laptops: a 2019 MacBook Pro and a KFocus M2 running Kubuntu. The lab creates millions of records and performs some aggregations on them. Before performing the aggregations the lab switches through four modes: no-cache, cache, traditional checkpoints, and non-eager checkpoints. Finally, I will go over the results of various tests I did over the last few weeks.
Details on the hardware used:
MacBook Pro (16-inch, 2019)
2.6GHz 6-Core Intel Core i7
16GB 2667 MHz DDR4
KFocus Model One
2.3GHz 8-Core Intel Core i7-10875H
64GB 3200 MHz DDR4
More information on the KFocus: https://kfocus.org/
Lab #100 is designed to work in an IDE. Lab #110 is designed to work on the command line, as in:
mvn -q exec:exec -Dargs="100000" -DXmx=4G
The repository includes the Excel sheet with all the measures for the various hardware I used.
This lab is based on chapter 16 of Spark in Action, 2nd edition (http://jgp.ai/sia).
Repository: https://github.com/jgperrin/net.jgp.b...
Lab: #100
GitHub link: https://github.com/jgperrin/net.jgp.b...
Lab: #110
GitHub link: https://github.com/jgperrin/net.jgp.b...
This event is supported by La French Tech Raleigh-Research Triangle ( / frenchtechraleigh .
#DataFriday