SVM Classifier: proper process I'm working on a classification system of mine, but am needing help with the proper process order. Specifically, I'm using LibSVM and a range of feature sets extracted from my data. I'm wondering, though, if it makes more sense to...
a) Perform feature selection via Forward Selection and using default LibSVM to evaluate results
b) Perform a grid search to find LibSVM parameters while using feature selected data
OR the opposite
a) Perform grid search to find LibSVM parameters while using all features
b) Perform feature selection via Forward Selection and using parameters found in grid search
To me, it makes more sense to reduce features with Forward Selection and default SVM parameters. Then, hone the parameters with the selected features. Unfortunately, maybe I'm just not searching the right keywords to find answers on this topic x_x
 A: The right order of things is unfortunately neither of the two. If you want to do feature selection, you ideally tune hyperparameters for each candidate set of features.
Doing feature selection without optimizing hyperparameters for each candidate set of features is a bad idea, because the hyperparameters can heavily impact model performance. Conversely, doing hyperparameter tuning in advance can lead to biased parameters based on some features, which will then inevitably result in those features being selected even though they may not be  the best.
In practice, however, SVM is quite robust against uninformative features, so it may not even be necessary to do feature selection at all. Secondly, grid search is a fairly poor search strategy for hyperparameter tuning (even though it is the standard approach). I recommend to use specialized tuning libraries like Optunity or Hyperopt, which provide disciplined search methods that will find much better sets of hyperparameters using far fewer candidate sets (= time).
