1
$\begingroup$

I am trying to do text classification using Naive Bayes. Before training, I would like to make feature selection in order to reduce the feature space dimension. In order to do so, I have thought of using a method that weights 2 filters for scoring the features and then select the top K features.

For example, let's suppose that I have Information Gain as the first filter, and "X" as the second filter. I would like to find the best weights a, b as follows:

score(feature) = a * Infogain(feature) + b * X(feature)

I guess one possible option should be trying with some values and see if the performance gets improved, but there are any other less costly methods ?

(for example, I thought of classifying a feature as good or bad using svm: a manual annotator classifies some features as good or bad ("label of the feature"), and Infogain(feature), X(feature) are used as "features or the feature" )

Thanks in advance...

Improve this question
$\endgroup$
7
  • $\begingroup$ Why don't you select those features that optimize the value of your performance measure? (Under cross-validation) $\endgroup$ – user765195 Apr 1 '12 at 15:53
  • $\begingroup$ Thanks. Hmm....So you are suggesting to try values for a,b and see what happens ? Isn't there any other way ? $\endgroup$ – kanzen_master Apr 1 '12 at 22:06
  • $\begingroup$ No, I'm not suggesting that. I'm asking, what is your performance measure? How would you quantify a classifier to be better than another one? Is it accuracy, AUC, precision at a particular recall? $\endgroup$ – user765195 Apr 1 '12 at 22:13
  • 1
    $\begingroup$ You can do something like this, though I've read a lot of arguments against doing it: do a backward feature elimination, with F-score as your measure. In other words, start with your full feature set, and see if eliminating a feature would improve your F-score under 10-fold cv. Keep repeating this process, until you get no improvement in your F-score. $\endgroup$ – user765195 Apr 2 '12 at 2:52
  • 1
    $\begingroup$ Let me see if I understand your question: you're looking for $a$ and $b$ such that using the linear combination you mention to rank features, and selecting the top $n$ features, F-score is maximized. Is that correct? $\endgroup$ – user765195 Apr 4 '12 at 4:21
2
$\begingroup$

Going back to your responses to my comments, I can think of two ways to do this:

  • Write $F$-Score as a function of your linear combination. Find $a$ and $b$ that maximize $F$-score by using an optimization method.
  • Do a grid search for $a$ and $b$ that maximize your $F$-score and test their validity by cross-validation.
Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks again for your time. Regarding your second option, I understand it is basically trying a bounded set of values for a,b and test the F score, then pick up the a,b values which result in a bigger F score. It looks ok to me but I would prefer another options... Regarding the first option, are you sure you can do that? I mean, a,b are the weights of "the features of the features", by combining it linearly you will get the score for a feature, not the F-Value of the classification. First option looks quite strange to me... $\endgroup$ – kanzen_master Apr 5 '12 at 5:48
  • 1
    $\begingroup$ The first option is not strange at all. It just might be difficult. $F$-score is clearly a function of $a$ and $b$ (for different $a$ and $b$, you get different features, and different $F$-scores). The challenge is to write this function in closed form. After that, it is just a search problem (say, using steepest-ascent.) But that's the same story in the second option. They're both search problems. It's just that in the first option, the search is a bit more guided (you go in the direction in which the gradient is maximum, which will hopefully get you to the global maximum.) $\endgroup$ – user765195 Apr 6 '12 at 3:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.