Train and Validation vs. Train, Test, and Validation I am embarking on a new job that will give me the opportunity to do some cool machine learning stuff.  I haven't touched this stuff on a deeper level since graduate school and I wanted to get some clarification on some concepts.
The way I was taught ML is that you split up your data (80/20) into training and validation datasets.  You fit your model to the 80% training split and get an error rate, loss, etc. through cross validation.  Then, you take the fitted model you constructed with the training data and pop in the 20% validation dataset to compare if the error rates, loss, etc are similar.  If so, the model is good.
I have been doing some research to refresh my knowledge, and I've been noticing 3-way splits now (training/test/validation) where the split is usually (70/20/10).  I'm so confused on how this 3-way split is different from the 2-way split I was taught in school.  Also, I'm pretty sure I've been interchanging test with validation when referring to the 2-way split methodology.
Can someone verify if my understanding of the 2-way split is correct and explain the difference between that and the 3-way split?
Thank you!
 A: In your two-way split, as you also mentioned, your validation set is actually your test set. In your way, you haven't mentioned about hyperparameter optimisation (HPO), but it's a key step in many machine learning algorithms. When you need HPO, you'll either need to have a separate validation set to tune the HPs or tune them using cross validation over the training set. In the end, the model is trained over the whole training dataset and tested over the test set.
In your ML algorithm, if you don't need to optimise HPs, you can obtain loss metrics using cross-validation over the training set as you did, but this could have been done by using the entire dataset as well, i.e. you have five 80-20 splits, and average the loss across folds. You don't need a two-level test.
A: The two methods you are describing are essentially the same thing. When you describe using cross validation, this is analogous to using a train test split just repeated multiple times. Train/validation/test and train/test with cross validation on the training set are exactly the same but using cross validation repeats for different splits of train/test.
A: In general, you need a split of your data set into test/training whenever there is danger of using information from the test set on the trainig set. The information might flow through you as a modeller. Let this sink in for a moment.
For example, if you build one model with one method, use a 80/20 split or cross validation. If you compare many methods on a 80/20 split you implicitly use information from performance on the test set in your modelling. Add a validation data set in this case. If you choose some of these models for hyperparameter optimization, you need an additional test test, or optimize models when they are built (this is hard). And so on.
In any case, keep the validation data set off the table for as long as possible and use it as little as possible.
A: “All models are wrong, but some are useful”. George E. P. Box
When we build a model, our main materials is data. But it is ultimate goal that probably will never be achieved is to use our model to different data and performs like we trained with our trained data. But we can obviously try and then comes this split.
Now I will not discuss the size of the split. But I will focus on whether we should split 2 ways or 3 ways. To me, it is unnecessary to split three part for the data you control. For example you have ECG data from 100 patients. You can train to see whether your model works by spliting 80/20 as train and test. If it performs well in test set, there is no need for validation as your test set was different tan your train set.
Question is when you need a validation set. For example, clinicians have extra 20 patients ECG data that were not used in you train and test set. You trained your model, tuned your parameters through your test set. But now you have completely different data which clinician has. They will try your model for validation. So the intuition can be, doctors want to have some model. They have 120 patients data. They give you 100 data. You split them as train and test. And after that doctors use your model in validation set.
In any competition for example kaggle, data probably divided this way.
