i want to make classification with naive Bayes. I have got about 100 Features. Numerical ones as well as categorical ones. Since i want only the most relevant ones to be included for the classification task i want to find them with some kind of feature elimination. My question now is the following: what is the method to use that for (paper/reference?!) and is this method implemented in some sort of software package. Since i use R i would especially prefer some R package.

Thanks in advance for your help!

  • $\begingroup$ Something in here might help you: stackoverflow.com/questions/20600388/… $\endgroup$ Jun 25, 2014 at 5:44
  • $\begingroup$ @ssdecontrol thank you! Indeed there was a hint that WEKA supports exhaustive search to select the subset of features that performs best. Does anybody know if R also supports such a exhaustive search? $\endgroup$ Jun 25, 2014 at 5:54
  • $\begingroup$ There's an R package called FSelector that seems to have a method for it, but I've never used it. It also hasn't been updated since February 2013. I'm surprised that R doesn't have a better-developed base of ML tools. Truth is, R supports anything you can program into it, it's just a matter of doing it. The only other thing I can offer is that you need to figure out a way to compute gain ratios efficiently; ML isn't something I know a whole lot about. Maybe some info here: stackoverflow.com/questions/17844520/… $\endgroup$ Jun 25, 2014 at 7:29
  • $\begingroup$ As of 2013 at least, it seems at least a few people are writing code from scratch in C++: www.iaeng.org/publication/WCE2013/WCE2013_pp1549-1554.pdf $\endgroup$ Jun 25, 2014 at 7:35
  • $\begingroup$ @ssdecontrol Thank you very much... i found exhaustive.search() in FSelector. This may do what i want to do. I will try to make some example with toy data and eventually post it as one kind of solution. $\endgroup$ Jun 25, 2014 at 8:07

2 Answers 2


There are two different routes you can take. The key word is 'relevance', and how you interpret it.

1) You can use a Chi-Squared test or Mutual information for feature relevance extraction as explained in detail on this link.

In a nutshell, Mutual information measures how much information the presence or absence of a particular term contributes to making the correct classification decision.

On the other hand, you can use the Chi Squared test to check whether the occurrence of a specific variable and the occurrence of a specific class are independent.

Implementating these in R should be straight-forward.

2) Alternatively, you can adopt a wrapper feature selection strategy, where the primary goal is constructing and selecting subsets of features that are useful to build an accurate classifier. This contrasts with 1, where the goal is finding or ranking all potentially relevant variables.

Note that selecting the most relevant variables is usually suboptimal for boosting the accuracy of your classifier, particularly if the variables are redundant. Conversely, a subset of useful variables may exclude many redundant, but relevant, variables.


The R package caret (**C**lassification **A**nd **R**Egression **T**raining) has built-in feature selection tools and supports naive Bayes. I figured I'd post this as an answer instead of a comment because I'm more confident about this one, having used it myself in the past.

  • 1
    $\begingroup$ yes you are right. I have already found the function rfe() in caret. But is this the best way to handle it? Are there other functions more recommended? Does anybody have further insights? $\endgroup$ Jun 25, 2014 at 8:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.