Text Preprocessing, NLP
The video provides theoretical explanation and coding exercises for the various preprocessing steps used on textual data. Which preprocessing steps are required on your data depends on the nature of tasks you intend to perform. More straightforward tasks like classifying between two types of class labels may require more preprocessing to reduce computational cost. The information lost in preprocessing is expected to have no or most minor effect on the model's performance. Tasks that are more complicated like text summarization, dialog agent, etc. would require the least preprocessing in order to retain as much information as possible. So, that the tasks being more complicated can be better understood. A relatively complex model and more training samples are required for the later tasks.
The text preprocessing steps discussed in the video include, handling stopwords, domain or format specific stopwords, stemming, lemmatization, punctuations, regular expressions, parts-of-speech tagging, sentence segmentation, and word tokenization.
#textpreprocessing #NLP #naturallanguageprocessing