2
$\begingroup$

I have a doubt regarding the cross validation approach and train-validation-test approach.

I was told that I can split a dataset into 3 parts:

  1. Train: we train the model.
  2. Validation: we validate and adjust model parameters.
  3. Test: never seen before data. We get an unbiased final estimate.

So far, we have split into three subsets. Until here everything is okay. Attached is a picture:

enter image description here

Then I came across the K-fold cross validation approach and what I don’t understand is how I can relate the Test subset from the above approach. Meaning, in 5-fold cross validation we split the data into 5 and in each iteration the non-validation subset is used as the train subset and the validation is used as test set. But, in terms of the above mentioned example, where is the validation part in k-fold cross validation? We either have validation or test subset.

When I refer myself to train/validation/test, that “test” is the scoring:

Model development is generally a two-stage process. The first stage is training and validation, during which you apply algorithms to data for which you know the outcomes to uncover patterns between its features and the target variable. The second stage is scoring, in which you apply the trained model to a new dataset. Then, it returns outcomes in the form of probability scores for classification problems and estimated averages for regression problems. Finally, you deploy the trained model into a production application or use the insights it uncovers to improve business processes.

Thank you!

I would like to cite this information from https://towardsdatascience.com/train-validation-and-test-sets-72cb40cba9e7

Training Dataset Training Dataset: The sample of data used to fit the model. The actual dataset that we use to train the model (weights and biases in the case of Neural Network). The model sees and learns from this data. Validation Dataset Validation Dataset: The sample of data used to provide an unbiased evaluation of a model fit on the training dataset while tuning model hyperparameters. The evaluation becomes more biased as skill on the validation dataset is incorporated into the model configuration. The validation set is used to evaluate a given model, but this is for frequent evaluation. We as machine learning engineers use this data to fine-tune the model hyperparameters. Hence the model occasionally sees this data, but never does it “Learn” from this. We(mostly humans, at-least as of 2017 😛 ) use the validation set results and update higher level hyperparameters. So the validation set in a way affects a model, but indirectly.

Test Dataset Test Dataset: The sample of data used to provide an unbiased evaluation of a final model fit on the training dataset.

The Test dataset provides the gold standard used to evaluate the model. It is only used once a model is completely trained(using the train and validation sets). The test set is generally what is used to evaluate competing models (For example on many Kaggle competitions, the validation set is released initially along with the training set and the actual test set is only released when the competition is about to close, and it is the result of the the model on the Test set that decides the winner). Many a times the validation set is used as the test set, but it is not good practice. The test set is generally well curated. It contains carefully sampled data that spans the various classes that the model would face, when used in the real world.

I Would like to say this: **Taking this into account, we still need the TEST split in order to have a good assement of our model. Otherwise we’re only training and adjusting parameters but never take the model to the battle field **

$\endgroup$
  • 2
    $\begingroup$ It is actually very simple given your description: cross-validation replaces the train/valid data split. You will still need an independent test set to get an unbiased final guess of the performance. $\endgroup$ – Michael M May 26 at 6:27
  • $\begingroup$ @MichaelM So, when we do train/validate/test on python or whatever, most of the times we are only working on our training data, hence our MSE or RMSE metric or you name it, is based on the train/validation split of the same dataset. If that’s the case, we are not appropriately assessing our model since we are not doing scoring, meaning, after having tuned the model and it works perfect, we don’t have that never-seen-before dataset called Test set, therefore we don’t have an accurate metric. We end up not knowin how our model works in the real world $\endgroup$ – NaveganTeX May 26 at 6:33
  • 1
    $\begingroup$ Check this video ~30:00 youtube.com/… $\endgroup$ – DuttaA May 26 at 6:34
  • 1
    $\begingroup$ Uhh...Yes? Validation set is not test set, as many k-fold implementation details are suggesting..test set is test set and once you touch it, its done...you have to report the result first time only (you get one chance for exam, if you fail you fail, so prepare accordingly in the k-fold dataset) I think you confused terminologies, most sources seem to get it wrong anyways.Check this answer: ai.stackexchange.com/questions/7756/… $\endgroup$ – DuttaA May 26 at 7:44
  • 1
    $\begingroup$ @naveganteX In your last example in the comments, you would be doing 2-fold CV on 80k rows. You don't apply CV to 40k rows, you train on 40k rows. Both folds in this case would be used once for training and once for testing. Note that 2-fold CV is usually not a particularly appealing choice, as your train set is quite small compared to the total sample size. More commonly, 10-fold CV is used. $\endgroup$ – Frans Rodenburg May 27 at 0:26
4
$\begingroup$

The general procedure of K fold Cross Validtion (CV) is:

  • Shuffle Dataset
  • Hold out some part of it ($~20\%$) whic will serve as your unbiased Test Set.
  • Select a set of hyper-parameters.
  • Divide the rest of your data into $K$-parts.
  • Use one part as validation set, rest as train set.
  • Your Validation performance (of given hyper-parameters) is determined/evaluated as the average of choosing each one of $K$ sets as CV set once (mathematically $\sum_KP(set^{(k)}) *(Performance) = \sum_K \frac{1}{K}*(Performance)$ (since randomly chosen).

Speaking in layman terms, assume you have a question bank and you have to report to others about your knowledge. You set out a certain number of questions as test (do not touch it except at the end). The rest you divide in $K$ parts and use one $(K-1)$ sets of question to train your knowledge (see both the question and answer) and the last one set to Validate your knowledge (solve questions, do not see answer), you do this for all sets choosing each time one set as the validation set. And finally, you take the test on the test set and report your knowledge.

$\endgroup$
  • 2
    $\begingroup$ +1. However, your equation for performance is only correct if all folds are the same size. If $\frac{n}{k}$ is not an integer, the last fold will not be of the same size and you should calculate performance weighted by sample size of folds instead. $\endgroup$ – Frans Rodenburg May 27 at 3:36
  • $\begingroup$ @FransRodenburg thanks for the tip...I actually did not know that.. $\endgroup$ – DuttaA May 27 at 6:35
  • $\begingroup$ @cbeleites could not understand what you are trying to say. $\endgroup$ – DuttaA May 28 at 9:17
3
$\begingroup$

Data splitting is only reliable if you have a very large data set, but since you mentioned $n = 100,000$ in the comments as an example, you should probably be fine. However, if your data set is small, you can get very different results with different splits. In that case, consider doing nested cross-validation instead.


The post you linked combines (normal, not nested) cross validation with a single random split, though. The entire procedure is as follows:

  1. Randomly divide the data set into a train and test set;
  2. Randomly divide your train set into $k$ parts;
  3. Choose your best model(s) by cross-validating on these $k$ parts:
    • Train on $k-1$ parts;
    • Evaluate performance on the remaining part;
    • Repeat until all parts are used once for evaluation;
  4. Retrain the best model(s) on the entire train set (or keep the models from step 3 for e.g. a majority vote);
  5. Evaluate the performance of your best model(s) (only a handful at most) on the test set.

The variance and bias estimates you obtain in step 5 are what you base your conclusions about.

The split in step 1 is up to you. Many use a 80/20 split, but if your data is large enough, you may be able to get away with a smaller test set. The split in step 2 should generally be as large as you can afford in terms of computation time. 10-fold CV is a common choice. You can even run step 2-3 multiple times and average the results. This is more robust against the different results you might have obtained from different random splits in step 2.

Finally, note that you should be careful with the use of the word unbiased. Cross-validation is still a form of internal validation and cannot account for the bias of this particular data set. The only way you could obtain an unbiased estimate would be through external validation (i.e. multiple data sets/studies/sources).

$\endgroup$
  • $\begingroup$ @cbeleites The procedure is a single cross-validation scheme with a holdout set for testing, as the OP asked for. However, for others who might come across this question I suggest to use nested cross-validation instead of data splitting into train/test when observations are limited. The two statements are separate, which is why I started a new paragraph. $\endgroup$ – Frans Rodenburg May 28 at 9:17
  • $\begingroup$ Ahh, OK. I misread your answer along the lines paragraph 1 giving totally correct advise that in many cases nested cross validation is better. Paragraph 2 this is how you do it [it still being nested CV]. May I suggest an edit to make this more clear? Please see if you approve. $\endgroup$ – cbeleites May 28 at 10:25
3
$\begingroup$

What is missing from the other answers so far is that cross validation is just a more sophisticated replacement for a single (aka hold out) split to split off part of a data set.

You can describe train/validate/test splitting (the first 3 lines of your diagram):

diagram explaining inner and outer splits

  1. Split the original set: split off the test set from outer training set and
  2. Split the outer training st: split off the validation set from out from inner training set.

Now, knowing what we want to split (blue in the diagram), we need to specify how each of these splits is done. In principle, we have the full range range of methods to produce (more or less) independent splits at our disposal, from

  • various resampling techniques (including cross validation) over
  • doing a random split once (aka hold out) to
  • getting truly new independent data, even according to a testing Design of Experiments (this could even be the external validation @FransRodenberg mentions)

These splitting methods (how) have different statistical and "data-logistic" properties that allow to choose what is good under which conditions.

  • If nothing else is said, the default is a single random split aka hold-out.

  • E.g. you may decide that the final test should not be done only on a data set randomly set aside from the original data, but should test the final model in various ways according to an experimental design that allows to interpret the test results with respect to multiple confounders and on cases acquired only after the model is finalized (fully trained) and no further parameter tuning takes place.
    So using such a data acquisition plan for the outer split.

  • You may also decide that for the inner split, cross validation should be used rather than a single random/hold out split, so that your hyperparameter optimization can profit from the lower variance uncertainty in the performance estimation and from the possibility to measure model stability.

  • etc.

So:

  • What the linked post describes is using cross validation for the innner split and hold out for the outer split.
    This is typically used if the tuning is done manually/interactively: you do whatever you think sensible within the outer training set. When you are finished, you "get the bill" by testing with the so far completely untouched test set.

  • From a statistics point of view, cross validation is better than a single random split (more precise at same bias, stability information possible), but at the cost of computation time.
    Thus, you can also replace the outer hold out split by a 2nd (outer) cross validation. (This works nicely if the hyperparameter tuning is automatic, but doesn't work if the tuning is done manually by a single human: it would be exceedingly tedious and human memory remembers the previously seen data thus breaks independence between the folds)
    If you look at the resulting code, you have 2 nested cross validation loops (outer split and inner split). This is why this technique is known as nested cross validation. Sometimes it is also called double cross validation.

    If you are concerned about computation time because of the nested loops with $(k_{inner} + 1) \cdot k_{outer} (+ 1)$ training steps, there are other resampling techiques that give you more freedom in choosing how many surrogate models are evaluated independently of the fraction of cases that is split off (it's of course not nested cross validation any more but nested name of actually used resampling technique).

  • For the sake of completeness, you could also decide to use hold out for the inner and cross validation for the outer split. While this is valid, it wouldn't have nice properties, though:

    The optimization/tuning done with the inner split requires high precision performance estimats to be stable. That's a clear indication for using a resampling technique (e.g. cross validation) also for the inner split => so use nested cross validation.

$\endgroup$
  • $\begingroup$ Awesome answer! Very clear, precise and concise. $\endgroup$ – NaveganTeX May 29 at 9:16
  • 1
    $\begingroup$ @naveganteX: thank you :-) $\endgroup$ – cbeleites May 29 at 12:12
0
$\begingroup$

The definition of training, validation and test sets may vary. Moreover, it is not always necessary to have three sets as you described. Sometimes, a training and a validation sets are enough.

In k-fold CV, you split your dataset into k different folds. You use k-1 folds to train your model and then you use the k-th fold to validate it. Now, to validate it, may even be replaced with to test it, since the k-th fold was not used for training. Then you repeat the process another k-1 times and the validation (or test) set will be different.

$\endgroup$
  • 2
    $\begingroup$ No, in K-fold cross-validation, you don't only have one fold left for validation (or testing). K-fold cross-validation involves randomly dividing the dataset into K groups or folds of approximately equal size. The first fold is kept for validation and the model is trained on K-1 folds. The process is then repeated K times and each time a different fold (different group of data points) are used for validation. You obtain K error values, which are then averaged to give you the cross-validation error. The testing set should be a completely separate set of data, not used for training. $\endgroup$ – AlexK May 26 at 5:45
  • 2
    $\begingroup$ But you end up using all this data for training. You can't then test the model on the same data. $\endgroup$ – AlexK May 26 at 5:47
  • 3
    $\begingroup$ You are mixing two things here: validation and testing. Yes, CV is beneficial for small datasets, but you are giving an impression to the OP that CV takes care of leaving out a completely separate set for scoring. CV or not CV, that has no influence on using a separate dataset for testing. $\endgroup$ – AlexK May 26 at 5:53
  • 2
    $\begingroup$ Recommended reading: stats.stackexchange.com/q/19048/241093 $\endgroup$ – AlexK May 26 at 6:06
  • 3
    $\begingroup$ @naveganteX, example: 100 samples, 50-25-25 train/validation/test split. With CV, the first 75 of that are used for training+validation and every sample of that 75 is used for both training and validation (that's what CV is). The remaining 25 of the test data is there no matter what. $\endgroup$ – AlexK May 26 at 6:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.