I built GPT-2 for $31.99
The difference between this video and the last GPT I made is that that one was a toy model. It was trained on a very small set of data (the works of Shakespeare) and it doesn’t get very good. This one is much bigger, and reaches the level of GPT-2. It required quite a lot more optimisation to train it.
This is Karpathy's tutorial: • Let's reproduce GPT-2 (124M)
The code: https://github.com/karpathy/build-nan...
My write up as I went: https://colab.research.google.com/dri...