3
$\begingroup$

I am getting confused with the concept of cross validation in machine learning. Suppose I have divided my training set in 3 folds: A,B,C. When I am training my model using 3 fold cv, I am training it as follows:

  1. On A,B (validating on C)
  2. On B,C (validating on A)
  3. On A,C (validating on B)

So essentially I am training my model on each fold (A,B,and C) twice. My question is how will this method be any different from training the model on entire training set (ABC) twice, since even with cv each fold is used twice for training? Any guidance on this confusion will be very helpful.

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ The trick is to evaluate your performance on the holdout data, which the training process doesn’t see (since it’s held out). Once you’ve tuned hyperparameters and selected them, you’d train on all of the data and make a final evaluation on yet another set of holdout data (often called a test set). $\endgroup$
    – Dave
    Sep 27, 2020 at 19:14
  • $\begingroup$ The use of cross-validation is when you have different potential models, or the same model but with different key-choices choices within the model (known as hyper-parameters). You use cross-validation to choose the model and to tune it, as an attempt to see how different choices behave for the validation sets of data (in a sense unseen data). Once you have chosen the form of the model and the hyper-parameters you can run it on the full set of training data, which may give for example you the best estimates of ordinary parameters within that model. $\endgroup$
    – Henry
    Sep 28, 2020 at 9:58
  • $\begingroup$ ... If you also held out a separate test set, you can then do a test of your final model on unseen data. But you can only do this once; if your test data leads you to refining your model, then it stops being testing and it instead becomes validation again. $\endgroup$
    – Henry
    Sep 28, 2020 at 10:01

2 Answers 2

Reset to default
12
$\begingroup$

In simplified terms you can think about it like this: suppose you are preparing pupils for an exam. You have three sets exercises from previous exams: A, B, and C. But the exercises in the upcoming exam will be different. Nonetheless you want to test how well students will do on the unseen exam tests, when trained on similar tests from the past.

Here is how you can do it: you give exercises A and B to student one, and after he learns to solve them you test his ability on C. For another student you give exercises A and C and test how well she does on the remaining set B. And for the third student you give B and C and test on A.

This way the scores obtained on the unseen tests, by all students, will be the average score you can reasonably assume those students will get in the upcoming exam. However if instead you show all your exercise sets: A, B, and C, to a student - then how will you able to test how well is he or she prepared? If you give the student the exercise he or she saw in training then the student might answer it perfectly from memory alone.

Same with classification methods. If you show them all the data - what data will you use to check how well "trained" they are? A simple silly method that memorises everything would score 100% on such a testing strategy. But the same method might be completely lost when presented with an unseen data point.

Improve this answer
$\endgroup$
10
  • 1
    $\begingroup$ (+1) I like this answer a lot :) $\endgroup$
    – Robert Long
    Sep 27, 2020 at 19:42
  • $\begingroup$ @Karolis: thx a lot for the great explanation! Clarifies my doubt :) $\endgroup$
    – Stats
    Sep 27, 2020 at 19:49
  • 1
    $\begingroup$ @Karolis: Sorry but I am probably over thinking on this. In your example, agreed that we can test for the accuracy of each student on the hold out test which will be missing if we train a student on ABC together as there is no hold out set. However when we test both the groups ( with cv and without) on the unseen test set, shouldn’t the combined knowledge of the three students be the same as the one who learned on ABC together? I mean in the both cases, the learning or training set was A B C which both the groups saw, and so should perform similarly on any exercise other than ABC? $\endgroup$
    – Stats
    Sep 27, 2020 at 20:03
  • $\begingroup$ @Stats Good follow up question, and no, you are not overthinking, you are thinking :) The thing to note in your example is how you measure the accuracy of the method. When you train on AB and take accuracy on C - you measure accuracy for the method on the data it hasn't seen yet. Then when you train the method on AC and measure on B - the method has no memory from its previous training, so again the accuracy is being measured on data it didn't see yet. This measurement allows you to predict the accuracy when the method will be trained on all - ABC and used on a new data point. Is this helpful? $\endgroup$
    – Karolis Koncevičius
    Sep 27, 2020 at 20:06
  • 1
    $\begingroup$ @Stats this is a bit of a different question now, but there are various strategies of getting the final model. The most common one (as far as I know) is to train the model on ABC together. The reasoning is like this: "by doing cross validation we saw how the model performs on unseen data when trained on 2/3 of the data we have. Therefore, after we train it on the full data its performance should probably be a bit better than what we saw in cross validation". $\endgroup$
    – Karolis Koncevičius
    Sep 27, 2020 at 20:27
1
$\begingroup$

Cross-validation is generally used to find parameters or model structure to ensure it works well on new, unseen data. If you train your model using all the data A, B and C without doing the cross-validation, you risk overfitting and ending up with a model that performs well during training, but doesn't generalize to new data.

By performing the training on two of the folds and testing its performance on the third, you can optimize for performance on the new data. Instead of maximizing predictive power on the training set, you choose parameters that maximize the performance on the unseen data.

While you would use the cross-validation to find out the optimal (hyper)parameters, you would still usually use all the data you have to train the model.

Example

One commonly used example of overfitting is choosing too high degree polynomial when fitting a curve.

Here, if you determine the correct degree for the polynomial by using 2/3 of the points for the fitting and 1/3 of the points for testing, you would see that the high degree polynomial generalizes less well than e.g. the straight line.

Overfitted polynomial

In real life (when dealing with prediction problems), we are most often concerned in making sure the model works with new data, rather than fitting perfectly to the training data.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ I like the pictures and sybolics for communicating ideas. If you look at answers by whuber or gung, you will see these done very well. $\endgroup$
    – EngrStudent
    Sep 28, 2020 at 11:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.