Using Spark and Hive - PART 1: Spark as ETL tool

Опубликовано: 16 Март 2026
на канале: Melvin L

47,320

208

Working with Spark and Hive

Part 1: Scenario - Spark as ETL tool
Write to Parquet file using Spark

Part 2: SparkSQL to query data from Hive
Read Hive table data from Spark

Create an External Table
Query the data from Hive
Add new data
Query the data from Hive

case class Person(name: String, age: Int, sex:String)
val data = Seq(Person("Jack", 25,"M"), Person("Jill", 25,"F"), Person("Jess", 24,"F"))

val df = data.toDF()

import org.apache.spark.sql.SaveMode
df.select("name", "age", "sex").write.mode(SaveMode.Append).format("parquet").save("/tmp/person")

//Add new data
val data = Seq(Person("John", 25,"M"))
val df = data.toDF()
df.select("name", "age", "sex").write.mode(SaveMode.Append).format("parquet").save("/tmp/person")

CREATE EXTERNAL TABLE person ( name String, age Int, sex String)
STORED as PARQUET
LOCATION '/tmp/person'