Spark Reading and Writing to Parquet Storage Format

Опубликовано: 30 Октябрь 2024
на канале: Melvin L

27,545

126

Spark: Reading and Writing to Parquet Format
--------------------------------------------------------------------------

Using Spark Data Frame save capability
Code/Approach works on both local HDD and in HDFS environments

Related video:
Introduction to Apache Spark and Parquet, • Apache Parquet & Apache Spark

Code for demo

case class Person(name: String, age: Int, sex:String)
val data = Seq(Person("Jack", 25,"M"), Person("Jill", 25,"F"), Person("Jess", 24,"F"))
val df = data.toDF()

import org.apache.spark.sql.SaveMode
df.select("name", "age", "sex").write.mode(SaveMode.Append).format("parquet").save("/tmp/person")

df.select("name", "age", "sex").write.partitionBy("sex").mode(SaveMode.Append).format("parquet").save("/tmp/person_partitioned/")

val sqlContext = new org.apache.spark.sql.SQLContext(sc)
val dfPerson = sqlContext.read.parquet("/tmp/person")