The tokenization pipeline
What happens when we call a tokenizer on some texts, and how does it compute the numbers it outputs?
This video is part of the Hugging Face course: http://huggingface.co/course
Open in colab to run the code samples:
https://colab.research.google.com/git...
Related videos:
The pipeline function: • The pipeline function
Learn more about the tokenizer algorithms: • Tokenizers Overview
Learn more about attention masks: • Batching inputs together (PyTorch)
Learn more about token type IDs: • Preprocessing sentence pairs (PyTorch)
Have a question? Checkout the forums: https://discuss.huggingface.co/c/cour...
Subscribe to our newsletter: https://huggingface.curated.co/