This is a rather general question. If the question is vague and hard to answer in a few lines, I'd be happy if someone just point me to some readings. Thanks in Advance.

I am working on a multi-class classification problem with a large datasets (>3000 variables) and I am looking to reduce the dimensionality. My first step is to using "filter" methods to evaluate the relevance of a single feature and the target variable to eliminate some very low score features.

I am planning on just using Mutual information as my relevance score, but as I read about it I realize there are more methods such as Fisher score, Gini index or simply correlation coefficients.

Q: Is there a better relevance measure for the given classifier I'd use? Eg. If I decide to use SVM, or decision tree as my classifier, would one of the relevance measure be more optimal than the others?

EDIT: I understand there are better methods (eg, some wrapper methods) in practice, but I am more interested in the theoretical analysis of how these filter methods mesh with the classifiers.


Traditionally, there are three families of feature selection methods (filter, wrapper and embedded). For your problem, if you already know what classifier are you going to use, you can just wrap the feature selection within. Or in other words - use the training accuracy of your classifier directly, instead of a filter measure (MI, Gini, correlation...).

A simple way of doing this would be:

  1. Split your training data into a train set and a validation set
  2. While there are untested feature subsets to test:

    • Choose the next subset of features to test - next_ft
    • Create a subset of train, using next_ft - subset_tr
    • Build a model using subset_tr. Evaluate it using the validation set.
    • If the performance of the model is the best one found so far, save next_ft as your current best feature subset - best_ft
  3. Use best_ft for your final model

For generating the subsets of features to test, you can just test all possible subsets (if the number of features is not too large) or use some kind of heuristic (local search/hill climbing, Sequential Forward/Backward Selection (SFS or SBS), Genetic Algorithms... there are many possibilities)

  • 1
    $\begingroup$ Thanks for explaining, but this is not exactly what I am asking. I understand the filter vs. wrapper difference. But I guess my questions is theoretically, is there a filter method (specifically between MI, Fisher scores, Gini index etc) that is better for some classifiers because the way the classifiers evaluate. $\endgroup$ – Yue Y Jun 9 '16 at 23:20
  • $\begingroup$ Not really. To the best of my knowledge, you may get better results with measures that correlate with the way in which the classifier works (e.g, i.i.r.c. CART uses Gini index when making the splits, so using the Gini index as a filter there might not be a bad idea). You'll probably need to follow that path. Another option - not a filter, but what I would use if concerned about the number of variables in your training set - would be to train a single linear model with L1 regularization, or a modern decision tree (e.g. Random Rorest) and use them to filter single variables with low (0) score $\endgroup$ – carrdelling Jun 10 '16 at 13:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.