Lemmatization has some obvious benefits in TF-IDF, e.g. it decreases the vocabulary size.
What are some other advantages, and what are some disadvantages to lemmatizing in the context of TF-IDF?
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.Sign up to join this community
It depends on what language are you working on and on what task are you trying to solve
Lemmatization would more likely help in topic modeling task, as suggested on: https://opendatagroup.github.io/data%20science/2019/03/21/preprocessing-text.html
Topic modeling, for example, relies on the distribution of content words, the identification of which is dependent on a string match between words, which is achieved by lemmatizing their forms so that all variants are consistent across documents
Lemmatization would more likely result in less accuracy in sentiment analysis, as suggested on: https://www.quora.com/Is-it-normal-to-get-better-accuracy-without-stemming-and-lemmatization-than-using-them-in-NLP-text-classification
For example, past tensed words could be judged as more negative / more likely to occur in negative sentences than present tense. The word
hated could be more negative than