I'm working on a classification system of mine, but am needing help with the proper process order. Specifically, I'm using LibSVM and a range of feature sets extracted from my data. I'm wondering, though, if it makes more sense to...

a) Perform feature selection via Forward Selection and using default LibSVM to evaluate results

b) Perform a grid search to find LibSVM parameters while using feature selected data

OR the opposite

a) Perform grid search to find LibSVM parameters while using all features

b) Perform feature selection via Forward Selection and using parameters found in grid search

To me, it makes more sense to reduce features with Forward Selection and default SVM parameters. Then, hone the parameters with the selected features. Unfortunately, maybe I'm just not searching the right keywords to find answers on this topic x_x


The right order of things is unfortunately neither of the two. If you want to do feature selection, you ideally tune hyperparameters for each candidate set of features.

Doing feature selection without optimizing hyperparameters for each candidate set of features is a bad idea, because the hyperparameters can heavily impact model performance. Conversely, doing hyperparameter tuning in advance can lead to biased parameters based on some features, which will then inevitably result in those features being selected even though they may not be the best.

In practice, however, SVM is quite robust against uninformative features, so it may not even be necessary to do feature selection at all. Secondly, grid search is a fairly poor search strategy for hyperparameter tuning (even though it is the standard approach). I recommend to use specialized tuning libraries like Optunity or Hyperopt, which provide disciplined search methods that will find much better sets of hyperparameters using far fewer candidate sets (= time).

  • $\begingroup$ This makes sense. Thanks for clarifying. I'm always up for reducing time! Thanks for the recommendation. I'm using MATLAB so it appears Optunity would work well. Thanks! $\endgroup$ Feb 5 '15 at 18:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.