Bay Area Spark Meetup: Adding Native SQL Support to Spark with Catalyst
Adding Native SQL Support to Spark with Catalyst
By Michael Armburst
Twitter: @michaelarmbrust
Slides: http://bit.ly/1mWnHGy
Host: Tagged, San Francisco
In this talk I'll describe Spark SQL, a new Alpha component that is part of the Spark 1.0 release. Spark SQL lets developers natively query data stored in both existing RDDs and external sources such as Apache Hive. A key feature of Spark SQL is the ability to blur the lines between relational tables and RDDs, making it easy for developers to intermix SQL commands that query external data with complex analytics. In addition to Spark SQL, I'll also talk about the Catalyst optimizer framework, which allows Spark SQL to automatically rewrite query plans to execute more efficiently.