Why LeCun Thinks Deep Learning Isn't Enough — Yann LeCun
We are now sponsored by Weights and Biases! Please visit our sponsor link: http://wandb.me/MLST
Are neural networks truly learning smooth, continuous data manifolds, or are they merely performing a high-resolution, piecewise linear slice-and-dice of the input space? In this episode, we unpack the mathematical reality of deep learning with Prof. Yann LeCun and Dr. Randall Balestriero from Meta AI.
We initially approached Balestriero and LeCun's paper, Learning in High Dimensions Always Amounts to Extrapolation, with heavy skepticism. However, this conversation fundamentally shifted our perspective on how neural networks operate. We explore why the traditional intuition of interpolation completely breaks down in high-dimensional spaces, meaning the classic convex hull definition of extrapolation is effectively broken when analyzing modern deep learning.
Prof. Yann LeCun is the Chief AI Scientist at Meta, a Turing Award laureate, and widely recognized as a foundational figure in deep learning. Dr. Randall Balestriero is a researcher at Meta AI whose recent work on the spline theory of deep neural networks provides a rigorous geometrical framework for understanding their underlying mechanics.
Main insights and topics discussed:
The spline theory of deep learning and how networks with ReLU activations recursively partition the ambient space into polyhedral convex cells.
Why neural networks are mathematically equivalent to compositions of linear functions, functioning more like locally sensitive hash tables than smooth geometric morphers.
The illusion of the homogeneous latent space, revealing that neural networks actually create input-specific affine transformations.
The curse of dimensionality and the mathematical impossibility of statistical generalization without severe inductive biases.
How the machine learning community's reliance on piecewise linear functions challenges the assumption that networks discover complex, smooth nonlinear realities.
---
TIMESTAMPS:
00:00:00 The Interpolation vs Extrapolation Debate
00:10:00 Weights and Biases Sponsorship
00:15:00 Spline Theory & Polyhedra in Neural Nets
00:25:00 The Piecewise Linear Reality of Deep Learning
00:35:00 Interpolative Representations & Feature Engineering
00:45:00 Convex Hulls and Local Generalization
00:55:00 The Curse of Dimensionality Visualized
01:05:00 Introducing Yann LeCun
01:10:00 LeCun on Why the Interpolation Dichotomy is Flawed
01:20:00 LeCun on Reasoning & Self-Supervised Learning
01:30:00 LeCun on Definitions of Interpolation & Attention
01:40:00 LeCun on Joint Embedding Architectures
01:50:00 LeCun on Energy Minimization & Model Predictive Control
02:01:00 Introducing Randall Balestriero & Redefining Interpolation
02:15:00 Balestriero on Why Deep Learning Actually Works
02:25:00 Balestriero on The Manifold Hypothesis & Separability
02:35:00 Balestriero on Spline Theory & Neural Decision Trees
02:45:00 Balestriero on MNIST vs ImageNet Dimensionality
02:55:00 Post-Interview Debrief: Interpolation & Reasoning
03:10:00 Post-Interview Debrief: Latent Space & The Lottery Ticket
---
REFERENCES:
Paper:
[00:03:10] Learning in High Dimensions Always Amounts to Extrapolation
https://arxiv.org/abs/2110.09485
[00:16:30] A Spline Theory of Deep Learning
https://arxiv.org/abs/1802.06975
[00:50:40] Interpolation of Sparse High-Dimensional Data
https://arxiv.org/abs/2006.13915
[01:42:10] BYOL (Bootstrap Your Own Latent)
https://arxiv.org/abs/2006.07733
[01:43:10] Barlow Twins
https://arxiv.org/abs/2103.03230
[01:44:10] VICReg
https://arxiv.org/abs/2105.04906
[02:35:10] Neural Decision Trees
https://arxiv.org/abs/1912.10098
Company:
[00:10:10] Weights & Biases
https://wandb.ai/
Website:
[00:25:30] TensorFlow Playground
https://playground.tensorflow.org/
Book:
[00:35:20] Deep Learning with Python (Francois Chollet)
https://www.amazon.com/Deep-Learning-...
[01:53:20] Thinking, Fast and Slow (Daniel Kahneman)
https://www.amazon.com/Thinking-Fast-...
Person:
[01:05:10] Yann LeCun
https://yann.lecun.com/
[01:21:30] Geoffrey Hinton
https://www.cs.toronto.edu/~hinton/
---
LINKS:
Full Transcript: https://app.rescript.info/share/98ksv...
Download PDF transcript: https://app.rescript.info/api/public/...
Patreon: / mlst
Discord: / discord
Interpolation of Sparse High-Dimensional Data [Dr. Thomas Lux]
https://tchlux.github.io/papers/tchlu...