Dynamic Priorities for Apache Spark Applications Resource Allocations (Michael Feiman)
Michael Feimen, a principal product architect at IBM, and Shinnosuke Okada a software engineer with Spark team at IBM discusses the importance of the dynamic service-level agreement (SLA) control for Spark applications that are implemented through resource allocation priorities.
These can be assigned to each Spark application at submission and then dynamically modified during the whole lifecycle of the application. The Spark application priority is translated by the scheduler to resource allocation adjustments for executors, and include either preemptive or non-preemptive release of resources. The priorities translation algorithm is determined by a scheduling policy to either define the order of executor allocations between applications or calculate the priority-weighed allocation shares among all submitted applications from the entire resource pool.
Learn more here: https://databricks.com/session/dynami...
Article you might like: https://databricks.com/session/model-...
About: Databricks provides a unified data analytics platform, powered by Apache Spark™, that accelerates innovation by unifying data science, engineering and business.
Read more here: https://databricks.com/product/unifie...
Connect with us:
Website: https://databricks.com
Facebook: / databricksinc
Twitter: / databricks
LinkedIn: / databricks
Instagram: / databricksinc Databricks is proud to announce that Gartner has named us a Leader in both the 2021 Magic Quadrant for Cloud Database Management Systems and the 2021 Magic Quadrant for Data Science and Machine Learning Platforms. Download the reports here. https://databricks.com/databricks-nam...