Arithmetic Transformers with Abacus Positional Embeddings | AI Paper Explained
In this video we dive into a recent research paper, titled: Transformers Can Do Arithmetic with the Right Embeddings. The paper introduces Abacus Embeddings, a new type of positional embeddings. Using Abacus Embeddings, the researchers were able to train state-of-the-art transformers for numbers addition, with impressive logical extrapolation capabilities - a model that was trained on 20-digit numbers, was able to achieve 99.1% accuracy over 100-digit numbers!
Watch the video to learn more.
Paper page - https://arxiv.org/abs/2405.17399
Code - https://github.com/mcleish7/arithmetic
-----------------------------------------------------------------------------------------------
✉️ Join the newsletter - https://aipapersacademy.com/newsletter/
👍 Please like & subscribe if you enjoy this content
-----------------------------------------------------------------------------------------------
Chapters:
0:00 Introduction
0:49 Abacus Embeddings
2:39 Logical Extrapolation
3:48 Input Injection
4:25 Recurrency