An introduction to Apache Parquet
In this video, we learn all about Apache Parquet, a column-based file format that's popular in the Hadoop/Spark ecosystem. We use pyarrow and parquet-cli to make sense of some Parquet files from the NYC Taxis dataset.
Resources:
Apache Parquet - https://parquet.apache.org/
The Parquet format specification - https://github.com/apache/parquet-format
Apache Arrow - https://arrow.apache.org/
Pandas commands for exporting DataFrames - https://pandas.pydata.org/docs/refere...
Parquet CLI - https://pypi.org/project/parquet-cli/
NYC Taxis Dataset - https://www1.nyc.gov/site/tlc/about/t...
GitHub Gist with code - https://gist.github.com/mneedham/1118...