Textbooks Are All You Need
"Textbooks Are All You Need" is a new paper from Microsoft research demonstrating the feasibility of training a powerful LLM for Python code with a relatively small training data set and number of model parameters.
Timestamps:
00:00 - Textbooks are all you need
01:46 - Headline results on HumanEval and MBPP
02:50 - Training & data details
05:05 - Filtering existing code datasets with a transformer
07:30 - The code exercises dataset
08:03 - Model architecture and training
08:30 - What does it cost to produce Phi-1?
10:16 - Data pruning experiments
11:16 - Implications for the commercial LLM ecosystem
11:42 - Limitations
13:20 - Closing thoughts
Paper link: https://arxiv.org/abs/2306.11644
Topics: #LLMs #ai #microsoft #phi-1
For related content:
Twitter: / samuelalbanie
Research lab: https://caml-lab.com/
personal webpage: https://samuelalbanie.com/
YouTube: / @samuelalbanie1
TikTok: / samuelalbanie
Instagram: / samuelalbanie
LinkedIn: / samuel-albanie
Discord server for filtir: / discord
(Optional) if you'd like to support the channel:
https://www.buymeacoffee.com/samuelal...
Acknowledgements:
Thanks to economist consultant Tom who pointed out the near-equivalence in pricing between Apple Vision Pros and the Phi-1 compute budget.