9
$\begingroup$

The wonderful libsvm package provides a python interface and a file "easy.py" that automatically searches for learning parameters (cost & gamma) that maximize the accuracy of the classifier. Within a given candidate set of learning parameters, accuracy is operationalized by cross-validation, but I feel like this undermines the purpose of cross-validation. That is, insofar as the learning parameters themselves can be chosen an a manner that might cause an over-fit of the data, I feel like a more appropriate approach would be to apply cross validation at the level of the search itself: perform the search on a training data set and then evaluate the ultimate accuracy of SVM resulting from the finally-chosen learning parameters by evaluation within a separate testing data set. Or am I missing something here?

$\endgroup$
10
$\begingroup$

If you learn the hyper-parameters in the full training data and then cross-validate, you will get an optimistically biased performance estimate, because the test data in each fold will already have been used in setting the hyper-parameters, so the hyper-parameters selected are selected in part because they suit the data in the test set. The optimistic bias introduced in this way can be unexpectedly large. See Cawley and Talbot, "On Over-fitting in Model Selection and Subsequent Selection Bias in Performance Evaluation", JMLR 11(Jul):2079−2107, 2010. (Particularly section 5.3). The best thing to do is nested cross-validation. The basic idea is that you cross-validate the entire method used to generate the model, so treat model selection (choosing the hyper-parameters) as simply part of the model fitting procedure (where the parameters are determined) and you can't go too far wrong.

If you use cross-validation on the training set to determine the hyper-parameters and then evaluate the performance of a model trained using those parameters on the whole training set, using a separate test set, that is also fine (provided you have enough data for reliably fitting the model and estimating performance using disjoint partitions).

$\endgroup$
  • $\begingroup$ Nice reference! $\endgroup$ – Andrew Jul 5 '11 at 16:09
  • $\begingroup$ In the second paragraph, how would you incorporate doing feature selection as well? Would it be ok to: i) do hyper parameter optimization as you said above (getting optimal hyper parameters) ii) run feature selection in another round of cross validation to get a set of top predictors (feature selection is run on training data partitioned into a subtraining and validation set using whatever resampling method used in hyper parameter optimization). iii) train a model with the top hyper parameter and top predictor set on full training data. Test on separate test set. $\endgroup$ – skim Jun 14 '18 at 15:40
  • $\begingroup$ to be a little more clear on ii) run feature selection in another round of cross validation to get a set of top predictors (training data is split into subtraining and validation set via resampling method used in hyper parameter optimization. then feature selection is run on subtraining data). $\endgroup$ – skim Jun 14 '18 at 15:48
  • $\begingroup$ Alternatively, could one do feature selection first via cross validation to get top feature set, then do hyper parameter tuning of whatever models of interest using the top feature set (as above in cross validation)? Then train the models with their optimal hyper parameters on full training data with only the already determined top feature set, and test on separate test set? $\endgroup$ – skim Jun 14 '18 at 15:55
0
$\begingroup$

I don't think cross-validation is misused in the case of LIBSVM because it is done on the testing data level. All it does is k-fold cross validation and search for the best parameter for RBF kernel. Let me know of you disagree.

$\endgroup$
  • $\begingroup$ The selection of the hyper-parameters should not involve the test data in any way, as this will lead to an optimistically biased performance estimate. Essentially choosing the hyper-parameters should be treated as an integral part of fitting the SVM, so the testing procedure needs to also test the error due to the selection of the hyper-parameters, see my paper that I reference in my answer to the question (it is open access). $\endgroup$ – Dikran Marsupial Feb 1 '13 at 8:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.