Non-Parametric Transformers | Paper explained
❤️ Become The AI Epiphany Patreon ❤️ ► / theaiepiphany
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬
In this video I do a deep dive of the "Self-Attention Between Datapoints: Going Beyond Individual Input-Output Pairs in Deep Learning" paper.
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬
✅ Non-Parametric Transformers paper: https://arxiv.org/abs/2106.02584
✅ Jay Alammar's BERT blog: http://jalammar.github.io/illustrated...
✅ My LinkedIn post (Judea Pearl): / aleksagordic_pearl-causality-intelligence-... (also check out my other posts I made related to this)
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬
⌚️ Timetable:
00:00 Key ideas of the paper
01:40 Abstract
02:55 Note on k-NN (non-parametric machine learning)
04:30 Data and NPT setup explained
06:15 NPT loss is inspired by BERT
08:20 A high-level architecture overview
11:30 NPT jointly learns imputation and prediction
12:50 Architecture deep dive (input embeddings, etc)
20:45 More details on the stochastic masking loss
23:30 Connections to Graph Neural Networks and CNNs
29:45 NPT achieves great results on tabular data benchmarks
34:25 NPT learns the underlying relational, causal mechanisms
39:40 NPT does rely on other datapoints
42:10 NPT attends to similar vectors
45:15 Conclusions
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬
💰 BECOME A PATREON OF THE AI EPIPHANY ❤️
If these videos, GitHub projects, and blogs help you,
consider helping me out by supporting me on Patreon!
The AI Epiphany ► / theaiepiphany
One-time donation:
https://www.paypal.com/paypalme/theai...
Much love! ❤️
Huge thank you to these AI Epiphany patreons:
Petar Veličković
Zvonimir Sabljic
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬
💡 The AI Epiphany is a channel dedicated to simplifying the field of AI using creative visualizations and in general, a stronger focus on geometrical and visual intuition, rather than the algebraic and numerical "intuition".
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬
👋 CONNECT WITH ME ON SOCIAL
LinkedIn ► / aleksagordic
Twitter ► / gordic_aleksa
Instagram ► / aiepiphany
Facebook ► / aiepiphany
👨👩👧👦 JOIN OUR DISCORD COMMUNITY:
Discord ► / discord
📢 SUBSCRIBE TO MY MONTHLY AI NEWSLETTER:
Substack ► https://aiepiphany.substack.com/
💻 FOLLOW ME ON GITHUB FOR COOL PROJECTS:
GitHub ► https://github.com/gordicaleksa
📚 FOLLOW ME ON MEDIUM:
Medium ► / gordicaleksa
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬
#nonparametric #transformers #relationalreasoning