11
$\begingroup$

My main question is with regards trying to understand how k-fold cross-validation fits in the context of having training/validation/testing sets (if it fits at all in such context).

Usually, people speak of splitting the data into a training, validation and testing set - say at a ratio of 60/20/20 per Andrew Ng's course - whereby the validation set is used to identify optimal parameters for model training.

However, if one wanted to use k-fold cross-validation in hope of obtaining a more representative accuracy measure when the amount of data is relatively small, what does doing k-fold cross-validation entail exactly in this 60/20/20 split scenario?

For instance, would that mean that we'd actually combine the training and testing sets (80% of the data) and do k-fold cross validation on them to obtain our accuracy measure (effectively discarding with having an explicit 'testing set'? If so, which trained model do we use a) in production, and b) to use against validation set and identify optimal training parameters? For instance, one possible answer for a and b is perhaps to use the best-fold model.

$\endgroup$
10
$\begingroup$

Cross-validation usually helps to avoid the need of a validation set.

The basic idea with training/validation/test data sets is as follows:

  1. Training: You try out different types of models with different choices of hyperparameters on the training data (e.g. linear model with different selection of features, neural net with different choices of layers, random forest with different values of mtry).

  2. Validation: You compare the performance of the models in Step 1 based on the validation set and select the winner. This helps to avoid wrong decisions taken by overfitting the training data set.

  3. Test: You try out the winner model on the test data just to get a feeling how good it performs in reality. This unravels overfitting introduced in Step 2. Here, you would not take any further decision. It is just plain information.

Now, in the case where you replace the validation step by cross-validation, the attack on the data is done almost identically, but you only have a training and a test data set. There is no need for a validation data set.

  1. Training: See above.

  2. Validation: You do cross-validation on the training data to choose the best model of Step 1 with respect to cross-validation performance (here, the original training data is repeatedly split into a temporary training and validation set). The models calculated in cross-validation are only used for choosing the best model of Step 1, which are all computed on the full training set.

  3. Test: See above.

$\endgroup$
  • 1
    $\begingroup$ Thanks! To confirm, in the CV context, one may have 80% train vs 20% test split. Then one may build a model on that 80% of the data and test against the 20% to get the accuracy. To try different model variations, one can do 10-fold CV on the training dataset (80% of data) - effectively training on 8% of the total data and testing against 72% of the total data in each fold. Based on the CV results, one can identify the optimal hyperparameter values and use them to build a new model trained on all training data (80% of the full dataset) and test against the remaining 20% test dataset. Correct? $\endgroup$ – blu Jul 17 '16 at 14:02
  • 3
    $\begingroup$ Yes, except that in each CV run, you would use 72% for training and 8% for validation ;-) $\endgroup$ – Michael M Jul 17 '16 at 14:30
  • $\begingroup$ Awesome response @MichaelM. I was reading up about nested cross-validation (NCV), and I'm having a hard time deciding if I should use it, or just do what you outlined for CV. And just so I understand it, NCV would be applied to step3. Instead of getting a 1 winner score, you get K winner scores (multiplying total runs by K, step 1-2 repeated K times with diff 80% train data), which you can then average. So questions: 1) is my understanding correct? 2) is it recommended to use NCV? $\endgroup$ – Aziz Javed Nov 1 '17 at 13:39
  • $\begingroup$ You are exactly right. Nested CV will help to get more reliable estimates than the "simple" approach outlined above. If time allows, it is definitively an option. Do you know on which data set the final model is calculated in nested CV? On the full? $\endgroup$ – Michael M Nov 1 '17 at 17:11
1
$\begingroup$

$K$-fold cross validation is a (re)sampling strategy like many others.

Splitting into training/validation/testing is also a sampling strategy.

You can substitute the training/validation for another sampling strategy. Then you would perform $K$-fold CV on 80% of the data and test on the remaining 20%.

You can also apply it to the testing part (this is what people call nested cross validation), where $K-1$ folds are used to training/validation and the remainder one to test, then you iterate this over folds.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.