Byte Pair Encoding Tokenization in NLP
#tokenization #transformers #nlp
Tokenization is the process of representing text into smaller meaningful lexical units. Byte Pair Encoding (BPE) is a popular subword-based tokenization algorithm used by state-of-the-art NLP models like RoBerta, BART, GPT, etc. In this video, we look into pros and cons of other methods and understand BPE through an example.
⏩ OUTLINE:
0:00 - Tokenization in NLP and it's types.
01:10 - Subword-level Tokenization
01:43 - Byte Pair Encoding (BPE) Algorithm
Enjoy reading articles? then consider subscribing to Medium membership, it is just 5$ a month for unlimited access to all free/paid content.
Subscribe now - / membership
*********************************************
If you want to support me financially which is totally optional and voluntary :) ❤️
You can consider buying me chai ( because I don't drink coffee :) ) at https://www.buymeacoffee.com/TechvizC...
*********************************************
⏩ IMPORTANT LINKS
Tokenization methods in NLP: • WordPiece Tokenization in NLP
Research Paper Summaries: • Simple Unsupervised Keyphrase Extraction u...
*********************************************
⏩ Youtube - / @techvizthedatascienceguy
⏩ LinkedIn - / prakhar21
⏩ Medium - / prakhar.mishra
⏩ GitHub - https://github.com/prakhar21
*********************************************
⏩ Please feel free to share out the content and subscribe to my channel - / @techvizthedatascienceguy
Tools I use for making videos :)
⏩ iPad - https://tinyurl.com/y39p6pwc
⏩ Apple Pencil - https://tinyurl.com/y5rk8txn
⏩ GoodNotes - https://tinyurl.com/y627cfsa
#techviz #datascienceguy #naturallanguageprocessing #machinelearning #ai
About Me:
I am Prakhar Mishra and this channel is my passion project. I am currently pursuing my MS (by research) in Data Science. I have an industry work-ex of 3+ years in the field of Data Science and Machine Learning with a particular focus on Natural Language Processing (NLP).