Since stemming chops of the word and gives back the stem and lemmatization focuses on root form of the word, individually both of them serves the purpose of reducing the length of the word tokens with certain advantages on one over the other. What is the ideal practice followed in industry? A - Both are used in subsequent steps during preprocessing B - Only stemming should be used C - Only Lemmatization should be used D - any other
Give relevant use cases for exemplification.
Background- stemming and lemmatization are both ways to shrink the size of the vocabulary space. By turning "running", "runner" and "runs" all into the stem or lemma "run", you can curb sparsity in your dataset. This is important as overly sparse data can lead to overfit (ie memorizing findings, not learning generalizable patterns.) Stemming is much faster as it's a fairly simple case-based algorithm. Lemmatizing is much more expensive yet doesn't offer an improvement proportionate to the increased computation time.
In either case, (or with neither method) you still need to vectorize your text inputs in some manner. TF-IDF (term frequency inverse document frequency) is a common means. It essentially says how import is a given word to a given document when contrasted with the frequency of the word occurring across all documents. Ex: If "horrible" is uncommon in a dataset of Yelp reviews but occurs three times in a specific review, then this word is assumed to play a more important role in this review than in others.
The last development in NLP text-preprocessing is using word vectors (see GloVe, Word2Vec for more info.) These methods map vocabulary to high dimensional space. And when a sequence of words is treated as the aggregation of word vectors, you get an embedding that is sometimes better than TF-IDF representation.
Short answer- go with stemming when the vocab space is small and the documents are large. Conversely, go with word embeddings when the vocab space is large but the documents are small. However, don't use lemmatization as the increased performance to increased cost ratio is quite low.
