# equations for removing noisy, indecisive or too rare features from Naive Bayes

I am looking for formulas/equations/criteria that identify which words from the feature set dictionary are noisy/indecisive or too rare.

For example, if the dictionary is let's say 5000 words and the corpus is 100000 documents and we are classifying an equally probable class, then a word that appears 50% of the time in bads and 50% of the times in goods is just noise.

For example, if the dictionary is let's say 5000 words and the corpus is 100000 documents, a word that appears once is too rare and hence useless.

For my problem, it is decidedly unbalanced.

Alas, I remember finding some simple equations for this but I cannot find them once again...

Thinking off the top of my head, I suppose I could use the Binomial proportion confidence interval as a criteria.

So if a word in totality appeared 10 times and within the bounds of the confidence interval, it would be considered inconclusive:

> p=0.5 #class distribution
> n=10
> c(p-2*sqrt(1/n*p*(1-p)),p+2*sqrt(1/n*p*(1-p)))
[1] 0.1837722 0.8162278


But I am sure that isn't the method I vaguely remember...

• 2010.telfor.rs/files/radovi/TELFOR2010_10_13.pdf I still need to review this paper. I also wonder if a concept from en.wikipedia.org/wiki/Cohen%27s_kappa could be used. Aug 2, 2017 at 17:24
• A standard method is to use Yates corrected Chi Square or Williams Corrected G test. They both lack power at small counts < 5. The Binomial test will work but Walds suffers < 30 so the more complicated formulas are warranted Aug 5, 2017 at 6:29
• In nlp.stanford.edu/IR-book/pdf/irbookonlinereading.pdf pages 271+, Mutual Information is another method. The frequency selection critera would imply the elimination of infrequent words as well. Nov 14, 2017 at 2:30

Be aware that sometimes rare words provide you very much information. Obviously, if a word appears once in the dataset, then it is useless. In many cases what people do is they first remove the stopwords and then just take some number of the most common words as features to consider in their algorithm. Given the fact that language data has long tails (there is lots of words that appear very rarely), it is more reasonable to take top $n$ words rather then figuring out how to cut the tail. One more thing that you should consider is that, instead of number of occurrences, you should rather look at metrics of importance as TF-IDF.