I am looking for formulas/equations/criteria that identify which words from the feature set dictionary are noisy/indecisive or too rare.

For example, if the dictionary is let's say 5000 words and the corpus is 100000 documents and we are classifying an equally probable class, then a word that appears 50% of the time in bads and 50% of the times in goods is just noise.

For example, if the dictionary is let's say 5000 words and the corpus is 100000 documents, a word that appears once is too rare and hence useless.

For my problem, it is decidedly unbalanced.

Alas, I remember finding some simple equations for this but I cannot find them once again...

Thinking off the top of my head, I suppose I could use the Binomial proportion confidence interval as a criteria.

So if a word in totality appeared 10 times and within the bounds of the confidence interval, it would be considered inconclusive:

> p=0.5 #class distribution
> n=10
> c(p-2*sqrt(1/n*p*(1-p)),p+2*sqrt(1/n*p*(1-p)))
[1] 0.1837722 0.8162278

But I am sure that isn't the method I vaguely remember...

  • $\begingroup$ 2010.telfor.rs/files/radovi/TELFOR2010_10_13.pdf I still need to review this paper. I also wonder if a concept from en.wikipedia.org/wiki/Cohen%27s_kappa could be used. $\endgroup$
    – Chris
    Aug 2, 2017 at 17:24
  • $\begingroup$ A standard method is to use Yates corrected Chi Square or Williams Corrected G test. They both lack power at small counts < 5. The Binomial test will work but Walds suffers < 30 so the more complicated formulas are warranted $\endgroup$
    – Chris
    Aug 5, 2017 at 6:29
  • $\begingroup$ In nlp.stanford.edu/IR-book/pdf/irbookonlinereading.pdf pages 271+, Mutual Information is another method. The frequency selection critera would imply the elimination of infrequent words as well. $\endgroup$
    – Chris
    Nov 14, 2017 at 2:30

1 Answer 1


Be aware that sometimes rare words provide you very much information. Obviously, if a word appears once in the dataset, then it is useless. In many cases what people do is they first remove the stopwords and then just take some number of the most common words as features to consider in their algorithm. Given the fact that language data has long tails (there is lots of words that appear very rarely), it is more reasonable to take top $n$ words rather then figuring out how to cut the tail. One more thing that you should consider is that, instead of number of occurrences, you should rather look at metrics of importance as TF-IDF.

Moreover, instead of ignoring some words you could consider some kind of dimensionality reduction like word2vec or GloVe (see also this blog entry) and use the transformed data as input for your algorithm. I know cases where this worked remarkably well even after transforming the noisy raw text (without any filtering, cleaning, stemming etc).

  • $\begingroup$ I should also add that the dictionary that I'm working with is actually on nominal codes and that's stop words do not apply. $\endgroup$
    – Chris
    Aug 2, 2017 at 17:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.