How would you describe cross-validation to someone without a data analysis background?
$\begingroup$
$\endgroup$
1
-
4$\begingroup$ This question is to help with any confusion over the site name proposal: meta.stats.stackexchange.com/questions/21/…. $\endgroup$– ShaneCommented Aug 18, 2010 at 13:23
Add a comment
|
5 Answers
$\begingroup$
$\endgroup$
3
Consider the following situation:
I want to catch the subway to go to my office. My plan is to take my car, park at the subway and then take the train to go to my office. My goal is to catch the train at 8.15 am every day so that I can reach my office on time. I need to decide the following: (a) the time at which I need to leave from my home and (b) the route I will take to drive to the station.
In the above example, I have two parameters (i.e., time of departure from home and route to take to the station) and I need to choose these parameters such that I reach the station by 8.15 am.
In order to solve the above problem I may try out different sets of 'parameters' (i.e., different combination of times of departure and route) on Mondays, Wednesdays, and Fridays, to see which combination is the 'best' one. The idea is that once I have identified the best combination I can use it every day so that I achieve my objective.
Problem of Overfitting
The problem with the above approach is that I may overfit which essentially means that the best combination I identify may in some sense may be unique to Mon, Wed and Fridays and that combination may not work for Tue and Thu. Overfitting may happen if in my search for the best combination of times and routes I exploit some aspect of the traffic situation on Mon/Wed/Fri which does not occur on Tue and Thu.
One Solution to Overfitting: Cross-Validation
Cross-validation is one solution to overfitting. The idea is that once we have identified our best combination of parameters (in our case time and route) we test the performance of that set of parameters in a different context. Therefore, we may want to test on Tue and Thu as well to ensure that our choices work for those days as well.
Extending the analogy to statistics
In statistics, we have a similar issue. We often use a limited set of data to estimate the unknown parameters we do not know. If we overfit then our parameter estimates will work very well for the existing data but not as well for when we use them in another context. Thus, cross-validation helps in avoiding the above issue of overfitting by proving us some reassurance that the parameter estimates are not unique to the data we used to estimate them.
Of course, cross validation is not perfect. Going back to our example of the subway, it can happen that even after cross-validation, our best choice of parameters may not work one month down the line because of various issues (e.g., construction, traffic volume changes over time etc).
-
2$\begingroup$ Technically, this is holdout validation but one can imagine extending the subway example to a cross-validation context. If it helps I will re-write the example and the rest of the text to be specific to cross-validation. $\endgroup$– user28Commented Aug 18, 2010 at 13:42
-
$\begingroup$ @srikant..the word overfitting as a word suggests me that something is "overly" done meaning ..more than required..so whats overly done ? Is it no of variables ? I am sorry I have been hearing the term overfitting many times and would take this oppurtunity to cleare this with you. $\endgroup$ Commented Nov 6, 2010 at 5:35
-
1$\begingroup$ I think this answer would be improved by adding the concept of folds, which is the key to how CV is different from holdout testing. In the example (using abbreviation M for Monday, T for Tuesday, ..., R for Thursday, ...): So you look at TWRF for the best combination and see how that works on M, then MWRF and see how it works on T, etc. So each day is used as a part of the problem four times and as the check to the problem once, but never as the problem and the check in the same round. $\endgroup$– WayneCommented Oct 10, 2016 at 21:10
$\begingroup$
$\endgroup$
3
I think that this is best described with the following picture (in this case showing k-fold cross-validation):
Cross-validation is a technique used to protect against overfitting in a predictive model, particularly in a case where the amount of data may be limited. In cross-validation, you make a fixed number of folds (or partitions) of the data, run the analysis on each fold, and then average the overall error estimate.
-
2$\begingroup$ It seems given discussions elsewhere on this site that k-fold cross validation is just one type of cross validation and describing it does not do the general job of describing what cross validation is. $\endgroup$ Commented Aug 18, 2010 at 14:31
-
3$\begingroup$ @drknexus: That's fair, but I mention that it's k-fold and I wanted to provide a visualization of the process to help explain it. $\endgroup$– ShaneCommented Aug 18, 2010 at 15:16
-
$\begingroup$ Quick clarification: is it the case that, nfold=4 means 25% for test set; nfold=5 means 20% for test set; nfold=10 means 10% for test set, and so forth: nfold = n means 100/n percent goes into the test set? $\endgroup$ Commented Oct 13, 2016 at 14:25
$\begingroup$
$\endgroup$
1
"Avoid learning your training data by heart by making sure the trained model performs well on independent data."
answered Aug 18, 2010 at 19:09
-
5$\begingroup$ This is motivation for cv, rather than a description of cv itself. $\endgroup$– jbaumsCommented Apr 21, 2016 at 1:18
$\begingroup$
$\endgroup$
0
Let's say you investigate some process; you've gathered some data describing it and you have build a model (either statistical or ML, doesn't matter). But now, how to judge if it is ok? Probably it fits suspiciously good to the data it was build on, so no-one will believe that your model is so splendid that you think.
First idea is to separate a subset of your data and use it to test the model build by your method on the rest of data. Now the result is definitely overfitting-free, nevertheless (especially for small sets) you could have been (un)lucky and draw (less)more simpler cases to test, making it (harder)easier to predict... Also your accuracy/error/goodness estimate is useless for model comparison/optimization, since you probably know nothing about its distribution.
When in doubt, use brute force, so just replicate the above process, gather few estimates of accuracy/error/goodness and average them -- and so you obtain cross validation. Among better estimate you will also get a histogram, so you will be able to approximate distribution or perform some non-parametric tests.
And this is it; the details of test-train splitting are the reason for different CV types, still except of rare cases and small strength differences they are rather equivalent. Indeed it is a huge advantage, because it makes it a bulletproof-fair method; it is very hard to cheat it.
$\begingroup$
$\endgroup$
Since you don't have access to the test data at the time of training, and you want your model to do well on the unseen test data, you "pretend" that you have access to some test data by repeatedly subsampling a small part of your training data, hold out this set while training the model, and then treating the held out set as a proxy to the test data (and choose model parameters that give best performance on the held out data). You hope that by randomly sampling various subsets from the training data, you might make them look like the test data (in the average behavior sense), and therefore the learned model parameters will be good for the test data as well (i.e., your model generalizes well for unseen data).
