BART Explained: Denoising Sequence-to-Sequence Pre-training
In this video we talk about how the Bidirectional and Auto-Regressive Transformer (BART) model works. Compared to other models like BERT or GPT, BART uses an encoder-decoder architecture that was trained using a combination of denoising autoencoder objectives and autoregressive language modeling.
References
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬
"BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension" paper: https://arxiv.org/abs/1910.13461
Related Videos
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬
Why Language Models Hallucinate: • Why LLMs Hallucinate
Grounding DINO, Open-Set Object Detection: • Object Detection Part 8: Grounding DINO, O...
Detection Transformers (DETR), Object Queries: • Object Detection Part 7: Detection Transfo...
Wav2vec2 A Framework for Self-Supervised Learning of Speech Representations - Paper Explained: • Wav2vec2 A Framework for Self-Supervised L...
Transformer Self-Attention Mechanism Explained: • Transformer Self-Attention Mechanism Visua...
How to Fine-tune Large Language Models Like ChatGPT with Low-Rank Adaptation (LoRA): • Low-Rank Adaptation (LoRA) Explained
Multi-Head Attention (MHA), Multi-Query Attention (MQA), Grouped Query Attention (GQA) Explained: • Multi-Head Attention (MHA), Multi-Query At...
LLM Prompt Engineering with Random Sampling: Temperature, Top-k, Top-p: • LLM Prompt Engineering with Random Samplin...
Contents
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬
00:00 - Intro
00:24 - BPE Encoding
00:50 - BERT vs GPT vs BART architectures
01:35 - BART text corruptions
02:52 - BART Fine-tuning
03:22 - Outro
Follow Me
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬
🐦 Twitter: @datamlistic / datamlistic
📸 Instagram: @datamlistic / datamlistic
📱 TikTok: @datamlistic / datamlistic
Channel Support
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬
The best way to support the channel is to share the content. ;)
If you'd like to also support the channel financially, donating the price of a coffee is always warmly welcomed! (completely optional and voluntary)
► Patreon: / datamlistic
► Bitcoin (BTC): 3C6Pkzyb5CjAUYrJxmpCaaNPVRgRVxxyTq
► Ethereum (ETH): 0x9Ac4eB94386C3e02b96599C05B7a8C71773c9281
► Cardano (ADA): addr1v95rfxlslfzkvd8sr3exkh7st4qmgj4ywf5zcaxgqgdyunsj5juw5
► Tether (USDT): 0xeC261d9b2EE4B6997a6a424067af165BAA4afE1a
#tokenization #llm #wordpiece #sentencepiece