On many tutorials about how to implement LSA, I see that stop words such as "and" are removed. I understand that we might find them in almost all kind of texts, but the repetition of link words in a text may have a meaning specific to this text.

More generally, how do I choose my stop words in LSA clustering ?

  • $\begingroup$ Aren't there standard dictionaries and agreed upon lists for stop words? For instance, Stanford's NLP Lab provides an English language dictionary for text mining where stop words are flagged. nlp.stanford.edu $\endgroup$ Apr 2 '16 at 10:42
  • $\begingroup$ The thing is that I'm working with a French dataset os standard list wouldn't work, moreover I read a paper showing that use of non-tailored stop words list decrease accuracy vastly : researchgate.net/publication/… $\endgroup$ Apr 2 '16 at 14:27
  • $\begingroup$ "Vastly" may be overblown. Here's a Python dictionary of stop words in multiple languages. pypi.python.org/pypi/stop-words/2014.5.26 I wouldn't rule using it out as a starting point. After all, you have to start somewhere, right? $\endgroup$ Apr 2 '16 at 14:32
  • $\begingroup$ thanks for the link! I'll start with it and adapt it as suggested by João in order to keep stop words which could give meaning. Thanks a lot $\endgroup$ Apr 2 '16 at 14:34

The stop words you should use depend on what text analysis you want to make. If you are interested in characteristics that are directly related to some commonly used stop words you shouldn't remove them.

Think of stop word removal as a form of feature selection using expert knowledge. The stop words list are created with words that usually don't have much information about the text.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.