# train / validation / test split

I understand that you typically use three different data sets (train/validation/test) to acquire an unbiased estimate of the performance measurement, because the models are tuned to fit for the train dataset (for parameter learning) and the validation dataset (for hyperparameter learning).

But, if my goal is to simply compare different models rather than acquiring unbiased estimate of the performance measurement, do I still need train/validation/test split? Wouldn't train/test split be enough?

• If your aim is to choose the best model to use and then test it, then that choice is in a sense a form of "hyperparameter learning" requiring a validation set. The test set should in theory only be used once – Henry Jun 19 '18 at 0:53
• What for you want to compare the models? What for would you use the models afterwards? Notice that without validation on external data you could easily train a model that would perfectly fit your training data (overfitting) and have error close to zero, while being totally unusable. – Tim Jun 19 '18 at 6:22
• Have you some idea about my comment below? – markowitz Oct 14 '19 at 15:36

• Exist some rules for determining the splits size ? For examples the simplest rule $1/3$ train $1/3$ validation and $1/3$ test; or $1/2$ train and $1/2$ test are reasonable? – markowitz Oct 9 '19 at 8:39