I am doing text analysis , I know before entering text to analysis , we need to do some pre-processing like return the word to its roots.

However, some words have different meaning like "meet" and "meeting" or "product" and "production" so if I return the word to its roots, the meaning would go.

Is there pro and cons for stemming? Can I skip stemming for certain words?

  • 1
    $\begingroup$ Unfortunately, neither stemmatization nor lemmatization can guarantee you the correct "meaning" of a word, these operations are not semantic analysis. $\endgroup$ – ttnphns Jan 7 '18 at 11:53
  • $\begingroup$ @ttnphns then why I should stem/lemmatize before analysis? $\endgroup$ – sara Jan 10 '18 at 4:29
  • 1
    $\begingroup$ Obviously, to bring various grammatical versions of a word to a single standard, so that, for example, "cat" and "cats" count as same word, or "beautiful" and "beautifully" as same word. This is grammatical, not semantic operation. $\endgroup$ – ttnphns Jan 10 '18 at 6:38

If you want to preprocess tokens, but don't want to use stemming, lemmatization is an alternative that collapses less words together.

For example in Python you can do this using nltk (you can also do it in R according to this answer)

>>> stemmer = nltk.stem.PorterStemmer()
>>> stemmer.stem('production')

But using Wordnet lemmatizer

>>> lemmatizer = nltk.stem.WordNetLemmatizer()
>>> lemmatizer.lemmatize('production')

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.