1
$\begingroup$

I understand that you typically use three different data sets (train/validation/test) to acquire an unbiased estimate of the performance measurement, because the models are tuned to fit for the train dataset (for parameter learning) and the validation dataset (for hyperparameter learning).

But, if my goal is to simply compare different models rather than acquiring unbiased estimate of the performance measurement, do I still need train/validation/test split? Wouldn't train/test split be enough?

$\endgroup$
  • 1
    $\begingroup$ If your aim is to choose the best model to use and then test it, then that choice is in a sense a form of "hyperparameter learning" requiring a validation set. The test set should in theory only be used once $\endgroup$ – Henry Jun 19 '18 at 0:53
  • 1
    $\begingroup$ What for you want to compare the models? What for would you use the models afterwards? Notice that without validation on external data you could easily train a model that would perfectly fit your training data (overfitting) and have error close to zero, while being totally unusable. $\endgroup$ – Tim Jun 19 '18 at 6:22
  • $\begingroup$ Have you some idea about my comment below? $\endgroup$ – markowitz Oct 14 at 15:36
1
$\begingroup$

Hyperparameters are integral part of a model. In case you want a fair comparison between two models, this includes choosing the hyperparameter values. If you use only train and test sets you can select the hyperparameter values which optimize the performance over the train set, but this will likely result in a model with poor generalization abilities, thus failing to perform well on the test set. Unless you propose a structured way to choose hyperparameter values which will still preserve the model generalization abilities - you need a validation set.

$\endgroup$
  • $\begingroup$ Exist some rules for determining the splits size ? For examples the simplest rule $1/3$ train $1/3$ validation and $1/3$ test; or $1/2$ train and $1/2$ test are reasonable? $\endgroup$ – markowitz Oct 9 at 8:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.