Cross-Validation in plain english?

Question

How would you describe cross-validation to someone without a data analysis background?

Answer 1

Consider the following situation:

I want to catch the subway to go to my office. My plan is to take my car, park at the subway and then take the train to go to my office. My goal is to catch the train at 8.15 am every day so that I can reach my office on time. I need to decide the following: (a) the time at which I need to leave from my home and (b) the route I will take to drive to the station.

In the above example, I have two parameters (i.e., time of departure from home and route to take to the station) and I need to choose these parameters such that I reach the station by 8.15 am.

In order to solve the above problem I may try out different sets of 'parameters' (i.e., different combination of times of departure and route) on MWF to see which combination is the 'best' one. The idea is that once I have identified the best combination I can use it every day so that I achieve my objective.

Problem of Overfitting

The problem with the above approach is that I may overfit which essentially means that the best combination I identify may in some sense may be unique to Mon, Wed and Fridays and that combination may not work for Tue and Thu. Overfitting may happen if in my search for the best combination of times and routes I exploit some aspect of the traffic situation on MWF which does not occur on Tue and Thu.

One Solution to Overfitting: Cross-Validation

Cross-validation is one solution to overfitting. The idea is that once we have identified our best combination of parameters (in our case time and route) we test the performance of that set of parameters in a different context. Therefore, we may want to test on Tue and Thu as well to ensure that our choices work for those days as well.

Extending the analogy to statistics

In statistics, we have a similar issue. We often use a limited set of data to estimate the unknown parameters we do not know. If we overfit then our parameter estimates will work very well for the existing data but not as well for when we use them in another context. Thus, cross-validation helps in avoiding the above issue of overfitting by proving us some reassurance that the parameter estimates are not unique to the data we used to estimate them.

Of course, cross validation is not perfect. Going back to our example of the subway, it can happen that even after cross-validation, our best choice of parameters may not work one month down the line because of various issues (e.g., construction, traffic volume changes over time etc).

Answer 2

I think that this is best described with the following picture (in this case showing k-fold cross-validation):

Cross-validation is a technique used to protect against overfitting in a predictive model, particularly in a case where the amount of data may be limited. In cross-validation, you make a fixed number of folds (or partitions) of the data, run the analysis on each fold, and then average the overall error estimate.

Answer 3

Let's say you investigate some process; you've gathered some data describing it and you have build a model (either statistical or ML, doesn't matter). But now, how to judge if it is ok? Probably it fits suspiciously good to the data it was build on, so no-one will believe that your model is so splendid that you think.
First idea is to separate a subset of your data and use it to test the model build by your method on the rest of data. Now the result is definitely overfitting-free, nevertheless (especially for small sets) you could have been (un)lucky and draw (less)more simpler cases to test, making it (harder)easier to predict... Also your accuracy/error/goodness estimate is useless for model comparison/optimization, since you probably know nothing about its distribution.
When in doubt, use brute force, so just replicate the above process, gather few estimates of accuracy/error/goodness and average them -- and so you obtain cross validation. Among better estimate you will also get a histogram, so you will be able to approximate distribution or perform some non-parametric tests.
And this is it; the details of test-train splitting are the reason for different CV types, still except of rare cases and small strength differences they are rather equivalent. Indeed it is a huge advantage, because it makes it a bulletproof-fair method; it is very hard to cheat it.

Answer 4

"Avoid learning your training data by heart by making sure the trained model performs well on independent data."

Answer 5

Since you don't have access to the test data at the time of training, and you want your model to do well on the unseen test data, you "pretend" that you have access to some test data by repeatedly subsampling a small part of your training data, hold out this set while training the model, and then treating the held out set as a proxy to the test data (and choose model parameters that give best performance on the held out data). You hope that by randomly sampling various subsets from the training data, you might make them look like the test data (in the average behavior sense), and therefore the learned model parameters will be good for the test data as well (i.e., your model generalizes well for unseen data).