1
$\begingroup$

I am learning about the use of cross-validation with grid-search to choose the best hyperparameter for SVM. The problem I came across is the references and examples of its application do not follow a singular standard.

On the one day, I have seen resource portraying the following steps:

  • 1a) Split data into training and test sets (say 50:50),
  • 1b) Use cross-validation and grid-search only on training set. Identify the hyperparameter set that gives the best performance,
  • 1c) Use the best hyperparameter set to train in the training set,
  • 1d) Lastly, use the trained model (from the best hyperparameter set) to make predictions in test set, and evaluate the performance from the test set.

Another way is the following:

  • 2a) Split data into validation, training, and test sets (say 20:40:40, respectively),
  • 2b) Use cross-validation and grid-search only on validation set. Identify the hyperparameter set that gives the best performance,
  • 2c) Use the best hyperparameter set to train in the training set,
  • 2d) Lastly, use the trained model (from the best hyperparameter set) to make predictions in test set, and evaluate the performance from the test set.

Is approach 2 more preferable than approach 1, or are they both accepted in research and academic settings? Approach 1 seems to be better than approach 2 because it doesn't require to expense the data into a separate validation set that the final SVM algorithm will never train with. Whereas, I have seen people citing approach 2 being more scientifically sound because it is less prone to overfitting in the training data. But the potential issue I see is, if you use a small validation set, the best hyperparameter set may be unreliable, but if you use a large validation set, you lose a lot of valuable data. Which should be used? Or does it depend?

$\endgroup$
  • $\begingroup$ Yes, options 2 is better, but it does require more data. So do it if you can afford it. $\endgroup$ – user2974951 Sep 16 at 7:06
  • $\begingroup$ Could you explain why? If doing parameter tuning in the training set will overfit in the training set, wouldn't doing parameter tuning in the validation set will overfit in the validation set? And now you use this best (from validation set) but potentially overfit parameter set to train the training set, wouldn't this still be a (or even a bigger) problem? $\endgroup$ – KubiK888 Sep 16 at 17:10
  • $\begingroup$ Yes that is of course a problem, hence why you would do that only if you can afford it, if you have a large enough sample, otherwise you may be better off with only 2 splits or none. $\endgroup$ – user2974951 Sep 17 at 11:30
  • $\begingroup$ Aside from data scarcity issue, what about overfitting the parameter based on the validation data? Why is it better to overfit in your validation data, then in your training data? $\endgroup$ – KubiK888 Sep 17 at 14:53
  • $\begingroup$ That is a good question. Have a look at stats.stackexchange.com/questions/19048/… $\endgroup$ – user2974951 Sep 18 at 10:53
1
$\begingroup$

AFAIK, I have never seen approach #2 used, and although of course I guess you can always point to someone who have used it, it is certainly far from usual, a valid reason being what you have already mentioned, i.e. that you end up not using a portion of your data (the validation set) for fitting your final model. Save the fact that the phrase "CV on the validation set" itself sounds weird (and with good reason).

Save for the splitting ratio (we usually use ratios like 70/30 or 80/20 - 50/50 sounds extreme), the approach #1 is the one to go, and the one most widely used.

$\endgroup$
  • $\begingroup$ Perhaps I am not using the term validation set as it is typically use. Let's call this a development set or preparation set. My underlying question is do I need to have a separate dataset other than the training set to tune the parameter (in order to prevent overfitting the training set). Or should I use the full or subsample of the training set to tune the parameters? $\endgroup$ – KubiK888 Sep 25 at 21:46
  • 1
    $\begingroup$ @KubiK888 As said, the approach #1 (without such a separate dataset) is OK and the most widely used $\endgroup$ – desertnaut Sep 25 at 22:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.