lakeFS Metadata Search Tutorial
𝗔𝗰𝗰𝗲𝗹𝗲𝗿𝗮𝘁𝗲 𝗗𝗮𝘁𝗮 𝗖𝘂𝗿𝗮𝘁𝗶𝗼𝗻 𝘄𝗶𝘁𝗵 𝗹𝗮𝗸𝗲𝗙𝗦 𝗠𝗲𝘁𝗮𝗱𝗮𝘁𝗮 𝗦𝗲𝗮𝗿𝗰𝗵
Metadata Search is a powerful feature that accelerates data curation by allowing you to explore, filter, and curate large-scale unstructured datasets using SQL queries.
What You'll Learn:
→ Dataset Curation Flow: Understand how Metadata Search fits into the data curation process, from dataset discovery to dataset definition.
→ The Problem with Unstructured Data: Learn why discovery is challenging with massive collections of unstructured files like images or videos, and how the lack of native metadata filtering slows down the process.
→ How Metadata Search Works: See how to automatically collect and index lakeFS object metadata, making it queryable with SQL.
• Query using system metadata (like object path, size, or timestamp) or user-defined metadata (like annotations or labels).
• Metadata is versioned alongside the data, ensuring queries return results for the exact version you choose.
• It relies on lakeFS-managed Iceberg tables, making the query interface compatible with Iceberg clients.
A Practical Example (Vision Language Model):
We'll walk through curating an image dataset for a Vision Language Model (VLM) using the blip3o dataset.
𝗗𝗮𝘁𝗮 𝗗𝗶𝘀𝗰𝗼𝘃𝗲𝗿𝘆: Explore metadata fields (like image entropy) and statistical properties using SQL and Python (DuckDB and PyIceberg) to understand data characteristics and define filtering thresholds.
𝗗𝗮𝘁𝗮𝘀𝗲𝘁 𝗗𝗲𝗳𝗶𝗻𝗶𝘁𝗶𝗼𝗻: Construct a curated dataset definition that includes a SQL query for filtering and a lakeFS tag to lock the query to an exact data version, ensuring reproducibility—all without copying any data.
By the end of this tutorial, you'll see how you can streamline data discovery, deepen exploration, and define curated datasets without building or maintaining separate metadata tracking and querying systems.
Want to try it out? Check the link below 👇
https://docs.lakefs.io/latest/dataman...