Cross Validation Vs Train Validation Test

Question

I have a doubt regarding the cross validation approach and train-validation-test approach.

I was told that I can split a dataset into 3 parts:

1. Train: we train the model.
2. Validation: we validate and adjust model parameters.
3. Test: never seen before data. We get an unbiased final estimate.

So far, we have split into three subsets. Until here everything is okay. Attached is a picture:

Then I came across the K-fold cross validation approach and what I don’t understand is how I can relate the Test subset from the above approach. Meaning, in 5-fold cross validation we split the data into 5 and in each iteration the non-validation subset is used as the train subset and the validation is used as test set. But, in terms of the above mentioned example, where is the validation part in k-fold cross validation? We either have validation or test subset.

When I refer myself to train/validation/test, that “test” is the scoring:

Model development is generally a two-stage process. The first stage is training and validation, during which you apply algorithms to data for which you know the outcomes to uncover patterns between its features and the target variable. The second stage is scoring, in which you apply the trained model to a new dataset. Then, it returns outcomes in the form of probability scores for classification problems and estimated averages for regression problems. Finally, you deploy the trained model into a production application or use the insights it uncovers to improve business processes.

Thank you!

I would like to cite this information from https://towardsdatascience.com/train-validation-and-test-sets-72cb40cba9e7

Training Dataset Training Dataset: The sample of data used to fit the model. The actual dataset that we use to train the model (weights and biases in the case of Neural Network). The model sees and learns from this data. Validation Dataset Validation Dataset: The sample of data used to provide an unbiased evaluation of a model fit on the training dataset while tuning model hyperparameters. The evaluation becomes more biased as skill on the validation dataset is incorporated into the model configuration. The validation set is used to evaluate a given model, but this is for frequent evaluation. We as machine learning engineers use this data to fine-tune the model hyperparameters. Hence the model occasionally sees this data, but never does it “Learn” from this. We(mostly humans, at-least as of 2017 😛 ) use the validation set results and update higher level hyperparameters. So the validation set in a way affects a model, but indirectly.

Test Dataset Test Dataset: The sample of data used to provide an unbiased evaluation of a final model fit on the training dataset.

The Test dataset provides the gold standard used to evaluate the model. It is only used once a model is completely trained(using the train and validation sets). The test set is generally what is used to evaluate competing models (For example on many Kaggle competitions, the validation set is released initially along with the training set and the actual test set is only released when the competition is about to close, and it is the result of the the model on the Test set that decides the winner). Many a times the validation set is used as the test set, but it is not good practice. The test set is generally well curated. It contains carefully sampled data that spans the various classes that the model would face, when used in the real world.

I Would like to say this: **Taking this into account, we still need the TEST split in order to have a good assement of our model. Otherwise we’re only training and adjusting parameters but never take the model to the battle field **

Answer 1

What is missing from the other answers so far is that cross validation is just a more sophisticated replacement for a single (aka hold out) split to split off part of a data set.

You can describe train/validate/test splitting (the first 3 lines of your diagram):

1. Split the original set: split off the test set from outer training set and
2. Split the outer training st: split off the validation set from out from inner training set.

Now, knowing what we want to split (blue in the diagram), we need to specify how each of these splits is done. In principle, we have the full range range of methods to produce (more or less) independent splits at our disposal, from

• various resampling techniques (including cross validation) over
• doing a random split once (aka hold out) to
• getting truly new independent data, even according to a testing Design of Experiments (this could even be the external validation @FransRodenberg mentions)

These splitting methods (how) have different statistical and "data-logistic" properties that allow to choose what is good under which conditions.

• If nothing else is said, the default is a single random split aka hold-out.

• E.g. you may decide that the final test should not be done only on a data set randomly set aside from the original data, but should test the final model in various ways according to an experimental design that allows to interpret the test results with respect to multiple confounders and on cases acquired only after the model is finalized (fully trained) and no further parameter tuning takes place.
  So using such a data acquisition plan for the outer split.

• You may also decide that for the inner split, cross validation should be used rather than a single random/hold out split, so that your hyperparameter optimization can profit from the lower variance uncertainty in the performance estimation and from the possibility to measure model stability.

• etc.

So:

• What the linked post describes is using cross validation for the innner split and hold out for the outer split.
  This is typically used if the tuning is done manually/interactively: you do whatever you think sensible within the outer training set. When you are finished, you "get the bill" by testing with the so far completely untouched test set.

• From a statistics point of view, cross validation is better than a single random split (more precise at same bias, stability information possible), but at the cost of computation time.
  Thus, you can also replace the outer hold out split by a 2nd (outer) cross validation. (This works nicely if the hyperparameter tuning is automatic, but doesn't work if the tuning is done manually by a single human: it would be exceedingly tedious and human memory remembers the previously seen data thus breaks independence between the folds)
  If you look at the resulting code, you have 2 nested cross validation loops (outer split and inner split). This is why this technique is known as nested cross validation. Sometimes it is also called double cross validation.

  If you are concerned about computation time because of the nested loops with $(k_{inner} + 1) \cdot k_{outer} (+ 1)$ training steps, there are other resampling techiques that give you more freedom in choosing how many surrogate models are evaluated independently of the fraction of cases that is split off (it's of course not nested cross validation any more but nested name of actually used resampling technique).

• For the sake of completeness, you could also decide to use hold out for the inner and cross validation for the outer split. While this is valid, it wouldn't have nice properties, though:

  The optimization/tuning done with the inner split requires high precision performance estimats to be stable. That's a clear indication for using a resampling technique (e.g. cross validation) also for the inner split => so use nested cross validation.

Answer 2

Data splitting is only reliable if you have a very large data set, but since you mentioned $n = 100,000$ in the comments as an example, you should probably be fine. However, if your data set is small, you can get very different results with different splits. In that case, consider doing nested cross-validation instead.

---

The post you linked combines (normal, not nested) cross validation with a single random split, though. The entire procedure is as follows:

1. Randomly divide the data set into a train and test set;
2. Randomly divide your train set into $k$ parts;
3. Choose your best model(s) by cross-validating on these $k$ parts:
   • Train on $k-1$ parts;
   • Evaluate performance on the remaining part;
   • Repeat until all parts are used once for evaluation;
4. Retrain the best model(s) on the entire train set (or keep the models from step 3 for e.g. a majority vote);
5. Evaluate the performance of your best model(s) (only a handful at most) on the test set.

The variance and bias estimates you obtain in step 5 are what you base your conclusions about.

The split in step 1 is up to you. Many use a 80/20 split, but if your data is large enough, you may be able to get away with a smaller test set. The split in step 2 should generally be as large as you can afford in terms of computation time. 10-fold CV is a common choice. You can even run step 2-3 multiple times and average the results. This is more robust against the different results you might have obtained from different random splits in step 2.

Finally, note that you should be careful with the use of the word unbiased. Cross-validation is still a form of internal validation and cannot account for the bias of this particular data set. The only way you could obtain an unbiased estimate would be through external validation (i.e. multiple data sets/studies/sources).

Answer 3

The general procedure of K fold Cross Validtion (CV) is:

• Shuffle Dataset
• Hold out some part of it ($~20\%$) whic will serve as your unbiased Test Set.
• Select a set of hyper-parameters.
• Divide the rest of your data into $K$-parts.
• Use one part as validation set, rest as train set.
• Your Validation performance (of given hyper-parameters) is determined/evaluated as the average of choosing each one of $K$ sets as CV set once (mathematically $\sum_KP(set^{(k)}) *(Performance) = \sum_K \frac{1}{K}*(Performance)$ (since randomly chosen).

Speaking in layman terms, assume you have a question bank and you have to report to others about your knowledge. You set out a certain number of questions as test (do not touch it except at the end). The rest you divide in $K$ parts and use one $(K-1)$ sets of question to train your knowledge (see both the question and answer) and the last one set to Validate your knowledge (solve questions, do not see answer), you do this for all sets choosing each time one set as the validation set. And finally, you take the test on the test set and report your knowledge.

Answer 4

The definition of training, validation and test sets may vary. Moreover, it is not always necessary to have three sets as you described. Sometimes, a training and a validation sets are enough.

In k-fold CV, you split your dataset into k different folds. You use k-1 folds to train your model and then you use the k-th fold to validate it. Now, to validate it, may even be replaced with to test it, since the k-th fold was not used for training. Then you repeat the process another k-1 times and the validation (or test) set will be different.