What is normalization?
The first step when tokenizing texts is called normalization. But what does this mean? This video will tell you all about it.
This video is part of the Hugging Face course: http://huggingface.co/course
Open in colab to run the code samples:
https://colab.research.google.com/git...
Related videos:
What is pre-tokenization? • What is pre-tokenization?
Training a new tokenizer: • Training a new tokenizer
Don't have a Hugging Face account? Join now: http://huggingface.co/join
Have a question? Checkout the forums: https://discuss.huggingface.co/c/cour...
Subscribe to our newsletter: https://huggingface.curated.co/