Cross validation confusion on the concept I am getting confused with the concept of cross validation in machine learning. Suppose I have divided my training set in 3  folds: A,B,C. When I am training my model using 3 fold cv, I am training it as follows:

*

*On A,B (validating on C)

*On B,C (validating on A)

*On A,C (validating on B)

So essentially I am training my model on each fold (A,B,and C) twice. My question is how will this method be any different from training the model on entire training set (ABC) twice, since even with cv each fold is used twice for training? Any guidance on this confusion will be very helpful.
 A: In simplified terms you can think about it like this: suppose you are preparing pupils for an exam. You have three sets exercises from previous exams: A, B, and C. But the exercises in the upcoming exam will be different. Nonetheless you want to test how well students will do on the unseen exam tests, when trained on similar tests from the past.
Here is how you can do it: you give exercises A and B to student one, and after he learns to solve them you test his ability on C. For another student you give exercises A and C and test how well she does on the remaining set B. And for the third student you give B and C and test on A.
This way the scores obtained on the unseen tests, by all students, will be the average score you can reasonably assume those students will get in the upcoming exam. However if instead you show all your exercise sets: A, B, and C, to a student - then how will you able to test how well is he or she prepared? If you give the student the exercise he or she saw in training then the student might answer it perfectly from memory alone.
Same with classification methods. If you show them all the data - what data will you use to check how well "trained" they are? A simple silly method that memorises everything would score 100% on such a testing strategy. But the same method might be completely lost when presented with an unseen data point.
A: Cross-validation is generally used to find parameters or model structure to ensure it works well on new, unseen data. If you train your model using all the data A, B and C without doing the cross-validation, you risk overfitting and ending up with a model that performs well during training, but doesn't generalize to new data.
By performing the training on two of the folds and testing its performance on the third, you can optimize for performance on the new data. Instead of maximizing predictive power on the training set, you choose parameters that maximize the performance on the unseen data.
While you would use the cross-validation to find out the optimal (hyper)parameters, you would still usually use all the data you have to train the model.
Example
One commonly used example of overfitting is choosing too high degree polynomial when fitting a curve.
Here, if you determine the correct degree for the polynomial by using 2/3 of the points for the fitting and 1/3 of the points for testing, you would see that the high degree polynomial generalizes less well than e.g. the straight line.

In real life (when dealing with prediction problems), we are most often concerned in making sure the model works with new data, rather than fitting perfectly to the training data.
