Positional Encoding | How LLMs understand structure
In this video, I have tried to have a comprehensive look at Positional Encoding, one of the fundamental requirements of transformer architectures. We start with understanding why PE is necessary through practical examples, explore how different encodings work, and dive deep into modern approaches like RoPE.
In this video, we cover:
Why transformers need positional encoding
Requirements for creating a positional encoder
Integer encoding and its limitations
Sinusoidal encoding from the original transformer paper
Rotary Position Encoding (RoPE) and its advantages
Inspiration has been heavily taken from the following blog:
https://huggingface.co/blog/designing...
To have a deep dive at transformers, consider reading my blog on the topic:
https://goyalpramod.github.io/blogs/T...
Chapters:
0:00 Introduction
0:30 Why we need positional Encoding
3:00 Integer Encoding
4:00 Sinusoidal Encoding
6:10 Rotary Position Encoding
7:10 RoPE Explained
8:45 RoPE Summary
Feel free to reach out to me from my various socials, I am more than happy to talk to people about stuff (not limited to AI/ML).