Lemmatization has some obvious benefits in TF-IDF, e.g. it decreases the vocabulary size.

What are some other advantages, and what are some disadvantages to lemmatizing in the context of TF-IDF?


It depends on what language are you working on and on what task are you trying to solve

Lemmatization would more likely help in topic modeling task, as suggested on: https://opendatagroup.github.io/data%20science/2019/03/21/preprocessing-text.html

Topic modeling, for example, relies on the distribution of content words, the identification of which is dependent on a string match between words, which is achieved by lemmatizing their forms so that all variants are consistent across documents

Lemmatization would more likely result in less accuracy in sentiment analysis, as suggested on: https://www.quora.com/Is-it-normal-to-get-better-accuracy-without-stemming-and-lemmatization-than-using-them-in-NLP-text-classification

For example, past tensed words could be judged as more negative / more likely to occur in negative sentences than present tense. The word hated could be more negative than hates.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.