How do Transformer Models keep track of the order of words? Positional Encoding
Transformer models can generate language really well, but how do they do it?
A very important step of the pipeline is the Positional Encoding step, which injects the order of the words into the LLM, in order for it to tell the difference between sentences with the same words, but in different order.
This is part of a series of videos on Large Language Models:
Video 1: The attention mechanism • The Attention Mechanism in Large Language ...
Video 2: The math behind the attention mechanism • The math behind Attention: Keys, Queries, ...
Video 3: Transformer models • What are Transformer Models and how do the...
Grokking Machine Learning Book:
https://www.manning.com/books/grokkin...
40% discount promo code: serranoyt
00:00 Introduction
03:01 How much to add to each word?
03:29 Some trigonometry
06:20 The formulas