7
$\begingroup$

Recently I have played with the pretrained GLOVE word embedding model for Twitter

http://nlp.stanford.edu/projects/glove/

I notice that common stopwords are existing in the model. That is, there is no stopword filtering before the training of the model.

I wonder if stopword filtering would improve performance in terms of:

  1. higher correlation (or cos-sim) between semantically similar words
  2. less noisy sum for aggregation of set of words, because I've heard the main problem of aggregation of word embedding is poor weighing on a significant portion of noisy words in the set.

Or does filtering stopwords give problems that I am not seeing?

$\endgroup$
  • 1
    $\begingroup$ In my personal experience with word embeddings and text classifications tasks I have always filtered out stopwords. In most cases they just add unnecessary noise to your model which can distort your machine-learning algorithm from finding a true signal. $\endgroup$ – RDizzl3 Mar 13 '16 at 0:54
  • 2
    $\begingroup$ I would argue for the opposite: By removing a fixed set of words you are making a priori assumptions about your corpus that might just be wrong. I know of work that specifically shows this to be the case for text classification tasks, but I am not aware of a conclusive paper on the issue of stopword filtering in neural embeddings. $\endgroup$ – fnl Mar 15 '16 at 9:22
  • 1
    $\begingroup$ @fnl Would you kindly share some pointers to reference, if you can recall? $\endgroup$ – Mai Mar 15 '16 at 12:50
  • 2
    $\begingroup$ For example: [Agarwal, S., & Yu, H. (2009). Automatically classifying sentences in full-text biomedical articles. Bioinformatics, 25(23), 3174–3180]. And there are other papers in bioinformatics that have concluded that stopword removal reduced classification performance. However, it is interesting that Glove's model is trained with stopwords, as the authors themselves reported boosted performance without them in earlier word representation work. [Huang et al. (2012). Improving word representations via global context and multiple word prototypes. ACL, 873–882.] $\endgroup$ – fnl Mar 16 '16 at 22:33
1
$\begingroup$

One common approach is to simply subsample the most common words in the corpus. This way they have less effect on the model but you don't have to completely get rid of them. It can also speed up training because you spend less time dealing with stopwords that don't carry all that much information compared to the amount of times they appear in the corpus.

https://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdf

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.