# Feature selection weighting 2 filters in Naive Bayes

I am trying to do text classification using Naive Bayes. Before training, I would like to make feature selection in order to reduce the feature space dimension. In order to do so, I have thought of using a method that weights 2 filters for scoring the features and then select the top K features.

For example, let's suppose that I have Information Gain as the first filter, and "X" as the second filter. I would like to find the best weights a, b as follows:

score(feature) = a * Infogain(feature) + b * X(feature)

I guess one possible option should be trying with some values and see if the performance gets improved, but there are any other less costly methods ?

(for example, I thought of classifying a feature as good or bad using svm: a manual annotator classifies some features as good or bad ("label of the feature"), and Infogain(feature), X(feature) are used as "features or the feature" )

• Why don't you select those features that optimize the value of your performance measure? (Under cross-validation) – user765195 Apr 1 '12 at 15:53
• Thanks. Hmm....So you are suggesting to try values for a,b and see what happens ? Isn't there any other way ? – kanzen_master Apr 1 '12 at 22:06
• No, I'm not suggesting that. I'm asking, what is your performance measure? How would you quantify a classifier to be better than another one? Is it accuracy, AUC, precision at a particular recall? – user765195 Apr 1 '12 at 22:13
• You can do something like this, though I've read a lot of arguments against doing it: do a backward feature elimination, with F-score as your measure. In other words, start with your full feature set, and see if eliminating a feature would improve your F-score under 10-fold cv. Keep repeating this process, until you get no improvement in your F-score. – user765195 Apr 2 '12 at 2:52
• Let me see if I understand your question: you're looking for $a$ and $b$ such that using the linear combination you mention to rank features, and selecting the top $n$ features, F-score is maximized. Is that correct? – user765195 Apr 4 '12 at 4:21

• Write $F$-Score as a function of your linear combination. Find $a$ and $b$ that maximize $F$-score by using an optimization method.
• Do a grid search for $a$ and $b$ that maximize your $F$-score and test their validity by cross-validation.
• The first option is not strange at all. It just might be difficult. $F$-score is clearly a function of $a$ and $b$ (for different $a$ and $b$, you get different features, and different $F$-scores). The challenge is to write this function in closed form. After that, it is just a search problem (say, using steepest-ascent.) But that's the same story in the second option. They're both search problems. It's just that in the first option, the search is a bit more guided (you go in the direction in which the gradient is maximum, which will hopefully get you to the global maximum.) – user765195 Apr 6 '12 at 3:02