Generalization in AI, with Dr. Dieuwke Hupkes
A must-listen episode with Dr. Dieuwke Hupkes https://dieuwkehupkes.nl/, a research scientist at @meta AI Research, where we dive into AI generalization, LLM robustness, and model evaluation in large language models.
We explore how LLMs handle grammar and hierarchy, how they generalize across tasks and languages, and what consistency tells us about AI alignment.
We also talk about Dieuwke’s journey from physics to NLP, the challenges of peer review, and sustaining a career in research—plus, how pole dancing helps with focus 💪
REFERENCES:
00:28 Dieuwke Hupkes - Google Scholar profile (https://scholar.google.ca/citations?u...)
24:32 A taxonomy and review of generalization research in NLP (https://www.nature.com/articles/s4225...)
41:28 What's in My Big Data (https://wimbd.apps.allenai.org)
49:49 GenBench workshop ( / @genbench-nlp , website - https://genbench.org)
52:30 Separating form and meaning: Using self-consistency to quantify task understanding across multiple senses (https://arxiv.org/abs/2305.11662)
52:38 From form(s) to meaning: Probing the semantic depths of language models using multisense consistency (https://direct.mit.edu/coli/article/5...)
01:02:44 MultiLoKo: a multilingual local knowledge benchmark for LLMs spanning 31 languages (https://arxiv.org/abs/2504.10356)
01:04:43 How much do language models memorize? (https://arxiv.org/abs/2505.24832)
ToC:
00:00 Introduction to Dieuwke Hupkes and Her Journey
05:15 Navigating Challenges in Research
07:17 The Peer Review Process: Insights and Frustrations
16:23 Being a Woman in AI: Representation and Challenges
19:57 Balancing Research and Personal Life
23:37 Exploring Consistency and Generalization in Language Models
33:31 Generalization Across Languages and Modalities
35:15 Exploring Generalization Taxonomy
40:55 Challenges in Evaluating Generalization
44:12 Data Contamination and Generalization
50:43 Consistency in Language Models
57:23 The Intersection of Consistency and Alignment
01:01:15 Current Research Directions
🎧 Subscribe to stay updated on new episodes spotlighting brilliant women shaping the future of AI.
WiAIR website:
♾️ https://women-in-ai-research.github.io
Follow us at:
♾️ LinkedIn: / women-in-ai-research
♾️ Bluesky: https://bsky.app/profile/wiair.bsky.s...
♾️ X (Twitter): https://x.com/WiAIR_podcast
#LLMs #AIgeneralization #LLMrobustness #AIalignment #ModelEvaluation #MetaAIResearch #WiAIR #WiAIRpodcast