Apache Spark and Kubernetes - Better together
Container technology has revolutionized the way production-level data centers are deployed
Most organizations are moving towards a containerized environment for big data workloads
Apache Spark also benefits greatly from the trend of containerized environments
Before we see how that works let’s first have a look at what Apache Spark is
*About Apache Spark*
Apache Spark is an open source analytics engine used to process huge amounts of data at a rapid pace
It can handle large chunks of data stored across multiple machines
Spark can do parallel processing of multiple datasets simultaneously which makes it a powerful tool utilising large amounts of computing power to complete tasks
With the use of an easy to understand API, Apache Spark removes a large burden for developers while while abstracting a lot of the work
Websites like Ebay use Spark for analytics and log transaction aggregation
Streaming service Netflix uses Apache Spark to process data to provide recommendations to users
It can run in standalone mode by installing Apache Spark framework and a JVM
For robust resource and cluster management, Spark requires an external scheduler
Users can choose among Apache Mesos, Docker Swarm, Hadoop YARN, or Kubernetes
Now let’s look at Spark running on Kubernetes
*Spark running on Kubernetes*
The Apache Spark framework provides APIs, which makes it compatible with Kubernetes
Spark apps can be submitted through a request which generates a Spark driver in a Kubernetes pod
The driver in turn creates executor pods and implements the application code
When Spark is deployed using Kubernetes, users can use multiple workloads together
Managing dependencies and updating the environment is a tedious process with Hadoop
With Kubernetes, all the dependencies can be managed independently in containers
Here are the advantages of Spark on Kubernetes
*Kubernetes advantages*
As stated in the beginning, containerization is a trend that many companies benefit from
Kubernetes allows users to quickly spin new machines and scale clusters on-demand
Apache Spark can have dependency issues which can be eliminated by packaging Spark applications in containers with their dependencies
Running Spark on Kubernetes can help achieve efficient use of resources which in turn saves a lot of time and resources, while simplifying management
According to a Data Mechanics article by Jean-Yves Stephan, Data Mechanics achieves 50 to 75% in cost saving for their customers when they migrate from other platforms to Spark on Kubernetes
[slide showing money and text: 50 to 75% cost savings]
Due to its unique advantages, it has been adopted by several global enterprises including Netflix, eBay, and Yahoo, for processing terabytes and petabytes of data
It is also one of the top open-source communities with a high number of contributors while also having an ecosystem with powerful monitoring add-ons
Some of the vendors that offer Spark services or Spark on Kubernetes includes
IBM Cloud Kubernetes Service, InsightEdge by GigaSpaces, Spot by NetApp and many others
For a leap ahead in how you manage Spark workloads, try Kubernetes
To know more about Spark on Kubernetes and related topics, please visit www.TwainTaylor.com