Word-based tokenizers
What is a character-based tokenizer, and what are the strengths and weaknesses of those tokenizers.
This video is part of the Hugging Face course: http://huggingface.co/course
Related videos:
Tokenizers overview: • Tokenizers Overview
Character-based tokenizers: • Character-based tokenizers
Subword-based tokenizers: • Subword-based tokenizers
Have a question? Checkout the forums: https://discuss.huggingface.co/c/cour...
Subscribe to our newsletter: https://huggingface.curated.co/