What Is In-Context Learning in Deep Learning?
How does In-context Learning work in LLMs (big and small)?
Table of Content
Introduction: 0:00
Definition of In-Context Learning: 0:59
Why is In-Context Learning Surprising?: 2:10
Bayesian Framework for In-Context Learning: 3:13
LLMs are Implementing Learning Algorithms for In-Context Learning: 7:59
Conclusion: 10:55
Cool Resource:
https://thegradient.pub/in-context-le...
http://ai.stanford.edu/blog/understan...
Interesting Papers:
Language Models are Few-Shot Learners: https://arxiv.org/pdf/2005.14165
An Explanation of In-context Learning as Implicit Bayesian Inference: https://arxiv.org/abs/2111.02080
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? https://arxiv.org/abs/2202.12837
In-context Learning and Induction Heads: https://transformer-circuits.pub/2022...
Transformers learn in-context by gradient descent https://arxiv.org/abs/2212.07677
Larger language models do in-context learning differently : https://research.google/blog/larger-l...
Really interesting topic! Hit me up in the comments if you want to discuss this in depth!
----
Join the newsletter for weekly AI content: https://yacinemahdid.com
Join the Discord for general discussion: / discord
----
Follow Me Online Here:
Twitter: / yacineaxya
GitHub: https://github.com/yacineMahdid
LinkedIn: / yacinemahdid
___
Have a great week! 👋