# Cross-Validation in plain english?

How would you describe cross-validation to someone without a data analysis background?

-
This question is to help with any confusion over the site name proposal: meta.stats.stackexchange.com/questions/21/…. –  Shane Aug 18 '10 at 13:23

Consider the following situation:

I want to catch the subway to go to my office. My plan is to take my car, park at the subway and then take the train to go to my office. My goal is to catch the train at 8.15 am every day so that I can reach my office on time. I need to decide the following: (a) the time at which I need to leave from my home and (b) the route I will take to drive to the station.

In the above example, I have two parameters (i.e., time of departure from home and route to take to the station) and I need to choose these parameters such that I reach the station by 8.15 am.

In order to solve the above problem I may try out different sets of 'parameters' (i.e., different combination of times of departure and route) on MWF to see which combination is the 'best' one. The idea is that once I have identified the best combination I can use it every day so that I achieve my objective.

Problem of Overfitting

The problem with the above approach is that I may overfit which essentially means that the best combination I identify may in some sense may be unique to Mon, Wed and Fridays and that combination may not work for Tue and Thu. Overfitting may happen if in my search for the best combination of times and routes I exploit some aspect of the traffic situation on MWF which does not occur on Tue and Thu.

One Solution to Overfitting: Cross-Validation

Cross-validation is one solution to overfitting. The idea is that once we have identified our best combination of parameters (in our case time and route) we test the performance of that set of parameters in a different context. Therefore, we may want to test on Tue and Thu as well to ensure that our choices work for those days as well.

Extending the analogy to statistics

In statistics, we have a similar issue. We often use a limited set of data to estimate the unknown parameters we do not know. If we overfit then our parameter estimates will work very well for the existing data but not as well for when we use them in another context. Thus, cross-validation helps in avoiding the above issue of overfitting by proving us some reassurance that the parameter estimates are not unique to the data we used to estimate them.

Of course, cross validation is not perfect. Going back to our example of the subway, it can happen that even after cross-validation, our best choice of parameters may not work one month down the line because of various issues (e.g., construction, traffic volume changes over time etc).

-
Technically, this is holdout validation but one can imagine extending the subway example to a cross-validation context. If it helps I will re-write the example and the rest of the text to be specific to cross-validation. –  user28 Aug 18 '10 at 13:42
Could the downvoters explain why they feel the answer does not help? –  user28 Aug 18 '10 at 20:12
I agree with Srikant. Would any downvoters please explain what's wrong. –  csgillespie Aug 18 '10 at 20:25
By the way, if something can be done to improve the answer, feel free to offer suggestions. I would be more than happy to edit the answer in response to reasonable, constructive feedback. –  user28 Aug 18 '10 at 20:37
@srikant..the word overfitting as a word suggests me that something is "overly" done meaning ..more than required..so whats overly done ? Is it no of variables ? I am sorry I have been hearing the term overfitting many times and would take this oppurtunity to cleare this with you. –  ayush biyani Nov 6 '10 at 5:35

I think that this is best described with the following picture (in this case showing k-fold cross-validation):

Cross-validation is a technique used to protect against overfitting in a predictive model, particularly in a case where the amount of data may be limited. In cross-validation, you make a fixed number of folds (or partitions) of the data, run the analysis on each fold, and then average the overall error estimate.

-
It seems given discussions elsewhere on this site that k-fold cross validation is just one type of cross validation and describing it does not do the general job of describing what cross validation is. –  rpierce Aug 18 '10 at 14:31
@drknexus: That's fair, but I mention that it's k-fold and I wanted to provide a visualization of the process to help explain it. –  Shane Aug 18 '10 at 15:16
This is very simple and useful –  Neil McGuigan Aug 18 '10 at 19:54

Let's say you investigate some process; you've gathered some data describing it and you have build a model (either statistical or ML, doesn't matter). But now, how to judge if it is ok? Probably it fits suspiciously good to the data it was build on, so no-one will believe that your model is so splendid that you think.
First idea is to separate a subset of your data and use it to test the model build by your method on the rest of data. Now the result is definitely overfitting-free, nevertheless (especially for small sets) you could have been (un)lucky and draw (less)more simpler cases to test, making it (harder)easier to predict... Also your accuracy/error/goodness estimate is useless for model comparison/optimization, since you probably know nothing about its distribution.
When in doubt, use brute force, so just replicate the above process, gather few estimates of accuracy/error/goodness and average them -- and so you obtain cross validation. Among better estimate you will also get a histogram, so you will be able to approximate distribution or perform some non-parametric tests.
And this is it; the details of test-train splitting are the reason for different CV types, still except of rare cases and small strength differences they are rather equivalent. Indeed it is a huge advantage, because it makes it a bulletproof-fair method; it is very hard to cheat it.

-
I don't think I'd call that "plain" English :) –  Neil McGuigan Aug 18 '10 at 19:53
I think it's not so bad, generally I don't plan to change it. –  mbq Aug 18 '10 at 20:12
This answer is plain english, but the punctuation screws it up. The "options" in the sentences in the second and third paragraphs (ie. the words in parentheses, or with slashes) make it really hard to read. You'd be better off just stating one case (lucky, more, easier), and then clarifying that it could go the other way. For the slashes, just replace them with commas or "or" where appropriate. They're painful as is. –  naught101 Mar 23 '12 at 3:02