5
$\begingroup$

For hyperparameter optimization I see two approaches:

  1. Splitting the dataset into train, validation and test, and optimize the hyperparameters based on the results of training on the train dataset and evaluating on the validation dataset, leaving the test set untouched for final performance estimation.

  2. Splitting the dataset into train and test, and optimize the hyperparameters using crossvalidation on the train set, leaving the test set untouched for final performance estimation.

So which approach is better?

$\endgroup$

1 Answer 1

6
$\begingroup$

Cross validation is more robust. So, in general, it is better. But, the marginal benefit you get decreases as dataset size increases. In small datasets, it's definitely suggested. On the other hand, it may not be the best choice due to computational complexity. For example, training might be very expensive, like in deep neural nets. In that case, a representative validation set is preferable over a statistical average of validation folds.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.