Tokenizers Overview
A general introduction to the different types of tokenizers.
This video is part of the Hugging Face course: http://huggingface.co/course
Related videos :
Word-based tokenizers: • Word-based tokenizers
Character-based tokenizers: • Character-based tokenizers
Subword-based tokenizers : • Subword-based tokenizers
Have a question? Check out the Hugging Face forum: https://discuss.huggingface.co/c/cour...
Don't have a Hugging Face account? Register now: http://huggingface.co/join