Data dictionaries, parquet, & Claude | Hadley Wickham | Data Science Lab
The Data Science Lab is a live weekly call. Register at pos.it/dslab! Discord invites go out each week on lives calls. We'd love to have you!
The Lab is an open, messy space for learning and asking questions. Think of it like pair coding with a friend or two. Learn something new, and share what you know to help others grow.
On this call, Libby Heeren is joined by Hadley Wickham, who walks through using data dictionaries with Claude Code to clean and document datasets effectively.
Hadley demonstrates a workflow using three files: a data cleaning script, a data dictionary in YAML format, and the final cleaned data as a Parquet file. He shows how Claude Code and MCP REPL can help generate and maintain data dictionaries that document what you know about your data, making it easier for both humans and AI agents to work with your datasets. Using the NYC elevators dataset as an example, he walks through data cleaning tasks like normalizing whitespace, handling missing values, fixing date formats, and investigating geocoding issues - all while keeping the data dictionary, cleaning script, and Parquet file in sync through git.
Hosting crew from Posit: Libby Heeren, Isabella Velasquez
Hadley Wickham's GitHub: https://github.com/hadley
Hadley Wickham's Bluesky: https://bsky.app/profile/hadley.nz
Hadley Wickham's LinkedIn: / hadleywickham
Resources mentioned in the video and chat:
🔗 MCP REPL: https://github.com/posit-dev/mcp-repl
🔗 Data Dictionary YAML Format Specification: https://github.com/hadley/data-dict.yaml
🔗 Parquet files in R for Data Science: https://r4ds.hadley.nz/arrow.html#sec...
🔗 Elevators Dataset Used in Demo: https://github.com/EmilHvitfeldt/elev...
🔗 Pointblank Package for Data Validation: https://posit-dev.github.io/pointblank/
🔗 Arrow R Book: https://arrowrbook.com/
🔗 Monaspace Font Family (with ligatures): https://monaspace.githubnext.com/
🔗 YAML Multiline Strings Reference: https://yaml-multiline.info/
🔗 UBC Course on Shiny with RAG and Parquet: https://ubc-mds.github.io/DSCI_532_vi...
🔗 Tom Scott Video on Timezones: • The Problem with Time & Timezones - Comput...
🔗 Falsehoods Programmers Believe About Names: https://www.kalzumeus.com/2010/06/17/...
🔗 UTC Is Enough for Everyone Right: https://zachholman.com/talk/utc-is-en...
🔗 Daniel Chen's Cherry Blossom Analysis: https://chendaniely.github.io/posts/2...
🔗 Project Drawdown Climate Impact Explorer: https://drawdown.org/explorer
🔗 Green Coding Bookdown Resource: https://bookdown.org/content/d1e53ac9...
🔗 IBM Green Coding Topics: https://www.ibm.com/think/topics/gree...
🔗 Secret Elevator in Central Park Article: https://undercovernyc.home.blog/2021/...
🔗 Artificial Cave Beneath Central Park: https://gizmodo.com/an-artificial-cav...
► Subscribe to Our Channel Here: https://bit.ly/2TzgcOu
Follow Us Here:
Website: https://www.posit.co
Hangout: https://pos.it/dsh
The Lab: https://pos.it/dslab
LinkedIn: / posit-software
Bluesky: https://bsky.app/profile/posit.co
Thanks for hanging out with us! 💛
Timestamps of Questions / Topics:
00:00 Introduction
03:38 "Can you talk a little bit about what MCP is?"
05:42 Introducing the elevators dataset
07:00 Creating initial data dictionary with Claude
09:03 "Are there any cases where the CSV format is actually a better choice than Parquet?"
09:50 "What font do you use?"
12:02 Adding context from the readme to the data dictionary
14:02 "What is a good way to store data dictionaries along with datasets?"
14:38 "Is this particular data dict YAML format useful for projects with only one table of data?"
17:45 "Is Claude also going to decide when it's a good time to make a commit?"
24:02 "Does MCP REPL work on a Windows machine and can one use other AI LLM for example ChatGPT with it?"
26:14 Converting date columns to proper date types
27:16 "Do you see a meaningful distinction between data dicts and data contracts?"
29:36 "How do you view your approach to data dictionaries and the development of pointblank?"
34:16 Eliminating placeholder values and using proper missing values
36:25 "Could you show off a diff of a Parquet file?"
38:25 Investigating geocodes and creating a map of elevators
40:20 Using a leaflet map to explore Central Park elevators
42:03 "Which model is Hadley using?"
43:55 Discussion of cost consciousness and environmental impact of LLMs
46:30 "Is there a way to quantify environmental and electrical costs?"
48:50 The mystery elevator in Central Park
54:23 "How do you know this is actually faster or more productive than just writing the code?"
55:45 The importance of deep knowledge of data in qualitative work