1
$\begingroup$

If I train a GNB/LDA/kNN/other classifier I would like to know, in the model built, how important are features to classify or which feature(s) drives the classifier.

For example in SVM models the importance of the feature is sometimes evaluated looking at weights magnitude, but for non-linear and generative models is more difficult to extract weights.

Do you know a book/paper that could be useful to solve my question?

$\endgroup$
1
$\begingroup$

One famous paper is Wrappers for feature subset selection by Kohavi and John (1997). It basically deals with the problem of removing the irrelevant features, which should provide an answer to your question. The approach is agnostic wrt the classifier used.

It's old, and it has almost 4k citations, so there is probably a wealth of follow-up work.

$\endgroup$
2
  • $\begingroup$ I'll take a look, but (looking at the title!) my question is after feature selection. So the scenario is basically after feature selection and cross-validation, I've the model and I would like to know which features are more important to the classification. $\endgroup$ – robbisg Jan 31 '13 at 16:09
  • 1
    $\begingroup$ They define two different approaches: a filter, which selects features independent of the classifier used (which wouldn't help you) and the wrapper approach, which basically runs the classifier (learner) with different feature subsets and evaluates the results to give the best subset. It doesn't literally tell you which features were actually dominant in the classification with the full set, but I would say that the subset that gives the best performance, are in some sense the most important. $\endgroup$ – Peter Jan 31 '13 at 16:31
1
$\begingroup$

I know of no book, but you can "probe" the behaviour for single variates: test what happens with the class scores if a certain input feature is varied, but the remaining are kept constant. For the nonlinear classifiers, make sure that your constant part and "probe pulse" are meaningful.

If you need a paper where feature influence on LDA was interpreted (not only coefficients, but also taking into account magnitude of input and irrelevant differences between models), we did this here: C. Beleites, K. Geiger, M. Kirsch, S. B. Sobottka, G. Schackert and R. Salzer: Raman spectroscopic grading of astrocytoma tissues: using soft reference information, Anal. Bioanal. Chem., 400 (2011), 2801 - 2816. The linked page has both links to the official web page and my manuscript.

$\endgroup$
1
$\begingroup$

The wrapper model popularized by Kohavi as mentioned @Peter would help in finding optimal features which are not necessarily relevant to your target labels. In the same paper, Kohavi states that "relevance does not imply optimal" and vice versa. In addition, the generally defined wrapper model does not rank the importance of your features.

You may follow one of the following solutions to rank the features selected by the wrapper model:

1- Rank the featuers using some filtering method as mRMR. Then, using forward selection you optimize your classifier and once the performance degrades you stop.

2- Select the featuers by the wrapper model and then, rank the selected ones by mRMR (opposite of 1).

3- Run the wrapper model over different folds of your data and then, you rank selected featuers based on frequency of being selected in different folds. An ensemble idea can follow this framework for featuer selection, also.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.