1
$\begingroup$

I am testing optimal subsets of features and I choose the SVM classifier. In the process, the training set is used for feature selection to train models with different subsets, validate the subset on the validation set, and test them on an unseen test set. whereas, I am optimizing the hyperparameters separately using all features and using the tuned hyperparameters for training the different subsets. I just want an opinion that the way I am conducting this experiment is right or not? if I am tuning the hyperparameters separately, do I need to have a validation set for validating the subsets? Or should i tune the hyperparameters for each subset or just globally?

$\endgroup$

1 Answer 1

1
$\begingroup$

It is possible to compute a bound on the leave-one-out error of the SVM (I've used the Radius-Margin bound and the Span bound and found they work quite well). This is often better than using a validation set as it leaves more data for training, and is computationally efficient.

If you are performing feature selection with the SVM, it may make generalisation performance worse rather than better, due to over-fitting the model/feature selection criterion (see also the paper by Ambroise and Maclachlan).

Personally I tend to tune the hyper-parameters from scratch each time I train it on a new sample of data (and adding or deleting features means it is a new sample). However, this can also lead to over-fitting the model selection criterion.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.