Apache Spark on Kubernetes: Best Practices & Performance Tuning
Speaker: Jean-Yves Stephan, Product Manager, Ocean for Spark, NetApp @JyStephan
Officially GA and production-ready, Spark-on-Kubernetes is the new standard deployment mode for Apache Spark applications (instead of Hadoop YARN and other cluster-managers). As the new kid on the block, there's a lot to learn about Spark and Kubernetes to realize all of its benefits.
In this session, we'll show you concrete technical tips and examples that you can leverage to be successful with Spark-on-Kubernetes, including:
Cluster autoscaling and Spark dynamic-allocation
Leveraging spot/preemptible nodes with Spark
Performance Optimizations (shuffle, S3, Disks and so on)
Monitoring best practices
Recent improvements and future works