Positional Encoding and Input Embedding in Transformers - Part 3
This is video no. 3 in the 5 part video series on Transformers Neural Network Architecture. This video is about the positional encoding and input embedding which takes care of converting the raw text input into numbers that are suited to be used by the Multi-Head Attention module in order to learn from data.
The first two videos in the series are here:
Part 1 - • Self-attention in deep learning (transform...
Part 2 - • Self-Attention in transfomers - Part 2
Transformers paper: https://arxiv.org/abs/1706.03762
Transformers paper abstract:
The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder-decoder configuration. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train. Our model achieves 28.4 BLEU on the WMT 2014 English-to-German translation task, improving over the existing best results, including ensembles by over 2 BLEU. On the WMT 2014 English-to-French translation task, our model establishes a new single-model state-of-the-art BLEU score of 41.8 after training for 3.5 days on eight GPUs, a small fraction of the training costs of the best models from the literature. We show that the Transformer generalizes well to other tasks by applying it successfully to English constituency parsing both with large and limited training data.
⌚️ ⌚️ ⌚️ TIMESTAMPS ⌚️ ⌚️ ⌚️
0:00 - Intro
0:22 - Input Embedding
1:37 - Input Embedding - Vocabulary
4:16 - Why Positional Encoding
3:46 - Positional Encoding
5:36 - Calculating Positional Encoding
9:04 - Thank YOU!
🛠 🛠 🛠 MY SOFTWARE TOOLS 🛠 🛠 🛠
✍️ Notion - https://affiliate.notion.so/aibites-yt
📹 OBS Studio for video editing - https://obsproject.com
📼 Manim for some animations - https://www.manim.community
🎵 My music - https://www.bensound.com and
📚 📚 📚 BOOKS I HAVE READ, REFER AND RECOMMEND 📚 📚 📚
📖 Deep Learning by Ian Goodfellow - https://amzn.to/3Wnyixv
📙 Pattern Recognition and Machine Learning by Christopher M. Bishop - https://amzn.to/3ZVnQQA
📗 Machine Learning: A Probabilistic Perspective by Kevin Murphy - https://amzn.to/3kAqThb
📘 Multiple View Geometry in Computer Vision by R Hartley and A Zisserman - https://amzn.to/3XKVOWi
MY KEY LINKS
YouTube: / aibites
Twitter: / ai_bites
Patreon: / ai_bites
Github: https://github.com/ai-bites
WHO AM I?
I am a Machine Learning Researcher / Practioner who has seen the grind of academia and start-ups equally. I started my career as a software engineer 15 years back. Because of my love for Mathematics (coupled with a glimmer of luck), I graduated with a Master's in Computer Vision and Robotics in 2016 when the now happening AI revolution just started. Life has changed for the better ever since.
#transformers #machinelearning #deeplearning #aibites