Parquet vs Avro
In this video we will cover the pros-cons of 2 Popular file formats used in the Hadoop ecosystem namely Apache Parquet and Apache Avro
Agenda:
Where these formats are used
Similarities
Key Considerations when choosing:
-Read vs Write Characteristics
-Tooling
-Schema Evolution
General guidelines
-Scenarios to keep data in both Parquet and Avro
Avro is a row-based storage format for Hadoop. However Avro is more than a serialisation framework its also an IPC framework
Parquet is a column-based storage format for Hadoop.
Both highly optimised (vs pain text), both are self describing , uses compression
If your use case typically scans or retrieves all of the fields in a row in each query, Avro is usually the best choice.
If your dataset has many columns, and your use case typically involves working with a subset of those columns rather than entire records, Parquet is optimized for that kind of work.
Finally in the video we will cover cases where you may use both file formats