Classifier feature importance If I train a GNB/LDA/kNN/other classifier I would like to know, in the model built, how important are features to classify or which feature(s) drives the classifier.
For example in SVM models the importance of the feature is sometimes evaluated looking at weights magnitude, but for non-linear and generative models is more difficult to extract weights.
Do you know a book/paper that could be useful to solve my question?
 A: I know of no book, but you can "probe" the behaviour for single variates: test what happens with the class scores if a certain input feature is varied, but the remaining are kept constant. For the nonlinear classifiers, make sure that your constant part and "probe pulse" are meaningful.
If you need a paper where feature influence on LDA was interpreted (not only coefficients, but also taking into account magnitude of input and irrelevant differences between models), we did this here: C. Beleites, K. Geiger, M. Kirsch, S. B. Sobottka, G. Schackert and R. Salzer: Raman spectroscopic grading of astrocytoma tissues: using soft reference information, Anal. Bioanal. Chem., 400 (2011), 2801 - 2816.  The linked page has both links to the official web page and my manuscript.
A: One famous paper is Wrappers for feature subset selection by Kohavi and John (1997). It basically deals with the problem of removing the irrelevant features, which should provide an answer to your question. The approach is agnostic wrt the classifier used.
It's old, and it has almost 4k citations, so there is probably a wealth of follow-up work.
A: The wrapper model popularized by Kohavi as mentioned @Peter would help in finding optimal features which are not necessarily relevant to your target labels. In the same paper, Kohavi states that "relevance does not imply optimal" and vice versa. In addition, the generally defined wrapper model does not rank the importance of your features.
You may follow one of the following solutions to rank the features selected by the wrapper model:
1- Rank the featuers using some filtering method as mRMR. Then, using forward selection you optimize your classifier and once the performance degrades you stop.
2- Select the featuers by the wrapper model and then, rank the selected ones by mRMR (opposite of 1).
3- Run the wrapper model over different folds of your data and then, you rank selected featuers based on frequency of being selected in different folds. An ensemble idea can follow this framework for featuer selection, also.
