GPT (nanoGPT) from a beginner’s perspective (Part 2 Final)
In this video I went through Karpathy's nanoGPT Code base, explaining the final part - Self-Attention.
nanoGPT is a character level implementation of a GPT (Generative Pre-trained Transformer) referencing the official Transformer from the Attention is All You Need paper.
Karpathy is my role model in the field of AI Research, he is a cofounder of Open AI and former Director of AI at Tesla.
My GPT Practice Repo
https://github.com/Jaykef/GPT-Practice
References
nanoGPT: https://github.com/karpathy/nanoGPT
Attension is All You Need Paper: https://arxiv.org/pdf/1706.03762.pdf
The Illustrated Transformer: https://jalammar.github.io/illustrate...
The Annotated Transformer: http://nlp.seas.harvard.edu/2018/04/0...
Techniques for training large neural networks: https://openai.com/research/technique...
The State of GPT: https://karpathy.ai/stateofgpt.pdf
#gpt #nanogpt #karpthy #ai #nlp #llm #openai #google