Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

To me, it seems that Hold-out validation is useless. That is, splitting the original dataset into two-parts (training and testing) and using the testing score as a generalization measure, is somewhat useless.

K-Fold seems to give better approximations of generalization (as it trains and tests on every point). So, why would we use the standard Hold-out validation? Or even talk about it?


share|improve this question
why do you think its useless? You can read Elements of Statistical learning theory section 7 for a formal analysis of its pro's and its con's. Statistically speaking, k-fold is better, but using a test set is not necessarily bad. Intuitively, you need to consider that a test set (when used correctly) is indeed a data set that has not been used at all at training. So its definitively useful in some sense to evaluate a model. Also, k-fold is super expensive, so hold out is sort of an "approximation" to what k-fold does (but for someone with low computational power). – Charlie Parker Dec 8 '15 at 19:38
Sure. From a theoretical perspective, K-fold is more precise but SLIGHTLY more computationally expensive. The question was: why not ALWAYS do K-fold cross validation? – zero Dec 8 '15 at 21:12
I see. I would argue that the reason is mostly always computational. K-fold approximates the generalization error better so from a statistical point of view K-fold is the method of choice I believe. Hold-out is much simpler to implement AND doesn't require training as many models. In practice, training a model can be quite expensive. – Charlie Parker Dec 9 '15 at 19:11
Right - but I think the "too computational expensive" argument is fairly frail. Almost all the time, we're aiming to develop the most accurate models. Yet there is this paradox where a lot of the experiments conducted in the literature only have a single hold-out validation set. – zero Dec 9 '15 at 19:55
up vote 9 down vote accepted

My only guess is that you can Hold-Out with three hours of programming experience; the other takes a week in principle and six months in practice.

In principle it's simple, but writing code is tedious and time-consuming. As Linus Torvalds famously said, "Bad programmers worry about the code. Good programmers worry about data structures and their relationships." Many of the people doing statistics are bad programmers, through no fault of their own. Doing k-fold cross validation efficiently (and by that I mean, in a way that isn't horribly frustrating to debug and use more than once) in R requires a vague understanding of data structures, but data structures are generally skipped in "intro to statistical programming" tutorials. It's like the old person using the Internet for the first time. It's really not hard, it just takes an extra half hour or so to figure out the first time, but it's brand new and that makes it confusing, so it's easy to ignore.

You have questions like this: How to implement a hold-out validation in R. No offense intended, whatsoever, to the asker. But many people just are not code-literate. The fact that people are doing cross-validation at all is enough to make me happy.

It sounds silly and trivial, but this comes from personal experience, having been that guy and having worked with many people who were that guy.

share|improve this answer
Maybe as someone who has majored in CS I have a slightly skewed view on this, but if you can implement hold-out validation correctly (which already means splitting the dataset into 2 parts and using one for training and the other for testing), the only thing you need to change is the ratio of the split and put the whole thing into a loop. It just seems hard to believe that this would be a big problem. – Voo Jun 25 '14 at 21:15
@Voo: in addition, being able to program is not enough here: you must understand the problem well enough to be able to judge for which confounders you need to account during your splitting procedure. See e.g.…. I think I see this kind of problems more often than "pure" coding problems (although one never knows: someone who's barely able to code a plain splitting of the rows in the data matrix will usually also make the higher-level mistake of not splitting e.g. at patient level) – cbeleites Jun 26 '14 at 12:21
Note also that you can do proper (e.g. patient/measurement day/...) hold-out splitting without any programming at all by separating the files the measurement instrument produces... – cbeleites Jun 26 '14 at 12:23
To the up-voters: note that I asked a separate question that questions my logic. – ssdecontrol Jul 20 '14 at 13:29

Hold-out is often used synonymous with validation with independent test set, although there are crucial differences between splitting the data randomly and designing a validation experiment for independent testing.

Independent test sets can be used to measure generalization performance that cannot be measured by resampling or hold-out validation, e.g. the performance for unknown future cases (= cases that are measured later, after the training is finished). This is important in order to know how long an existing model can be used for new data (think e.g. of instrument drift). More generally, this may be described as measuring the extrapolation performance in order to define the limits of applicability.

Another scenario where hold-out can actually be beneficial is: it is very easy to ensure that training and test data are properly separated - much easier than for resampling validation: e.g.

  1. decide splitting (e.g. do random assignment of cases)
  2. measure
  3. measurement and reference data of the training cases => modeling\ neither measurements nor reference of test cases is handed to the person who models.
  4. final model + measurements of the held-out cases => prediction
  5. compare predictions with reference for held-out cases.

Depending on the level of separation you need, each step may be done by someone else. As a first level, not handing over any data (not even the measurements) of the test cases to the modeler allows to be very certain that no test data leaks into the modeling process. At a second level, the final model and test case measurements could be handed over to yet someone else, and so on.

Yes, you pay for that by the lower efficiency of the hold-out estimates compared to resampling validation. But I've seen many papers where I suspect that that the resampling validation does not properly separate cases (in my field we have lots of clustered/hierarchical/grouped data).

I've learned my lesson on data leaks for resampling by retracting a manuscript a week after submission when I found out that I had a previously undetected (by running permutation tests alongside) leak in my splitting procedure (typo in index calculation).

Sometimes hold-out can be more efficient than finding someone who is willing to put in the time to check the resampling code (e.g. for clustered data) in order to gain the same level of certainty about the results. However, IMHO it is usually not efficient to do this before you are in the stage where you anyways need to measure e.g. future performance (first point) - in other words, when you anyways need to set up a validation experiment for the existing model.

OTOH, in small sample size situations, hold-out is no option: you need to hold out enough test cases so that the test results are precise enough to allow the needed conclusion (remember: 3 correct out of 3 test cases for classification means a binomial 95% confidence interval that ranges well below 50:50 guessing!) Frank Harrell would point to the rule of thumb that at least ca. 100 (test) cases are needed to properly measure a proportion [such as the fraction of correctly predicted cases] with a useful precision.

Update: there are situations where proper splitting is particularly hard to achieve, and cross validation becomes unfeasible. Consider a problem with a number of confounders. Splitting is easy if these confounders are strictly nested (e.g. a study with a number of patients has several specimen of each patient and analyses a number of cells of each specimen): you split at the highest level of the sampling hierarchy (patient-wise). But you may have independent confounders which are not nested, e.g. day-to-day variation or variance caused by different experimenters running the test. You then need to make sure the split is independent for all confounders on the highest level (the nested confounders will automatically be independent). Taking care of this is very difficult if some confounders are only identified during the study, and designing and performing a validation experiment may be more efficient than dealing with splits that leave almost no data neither for training nor for testing of the surrogate models.

share|improve this answer
I wish I could give more than +1 for this very thorough answer. I particularly liked you mentioning your issue with a data leak as it effectively illustrates that it can be far from trivial to rule out such problems, even for experts. This is a good reality check! – Marc Claesen Jun 26 '14 at 11:17
Aren't you begging the question? Yes, splitting is hard, due to confounders, but it's hard regardless of whether you're doing a single hold-out validation or k-fold cross-validation, isn't it? (Thanks for an insightful answer regardless!) – Nils von Barth Feb 1 at 5:51
@NilsvonBarth: I don't see how my arguments are circular: the OP asks "why [at all] use hold-out validation", and I give a bunch of practical reasons. The statistically most efficient use of a limited number of cases is not always the most important property of the study design. (Though in my experience it often is, due to extremely limited case numbers: I'm far more often advising for repeated/iterated k-fold CV instead of hold-out). For some confounders physical splitting is possible and easy - and a very efficient way to prevent sneak-previews. Who knows whether we'll find that doubly ... – cbeleites Feb 1 at 20:27
blinded statistical data analysis may be needed against too many false positive papers at some point? – cbeleites Feb 1 at 20:28
@cbeleites Thanks for clarifying! Summary of "cross-validation is most efficient (statistically) but that's not always most important, and hold-out allows guaranteed independence/prevents peeking" is useful (as are the rest of the details). – Nils von Barth Feb 2 at 3:37

It might be useful to clear up the terminology a little. If we let $k$ be some integer less than (or equal to) $n$ where $n$ is the sample size and we partition the sample into $k$ unique subsamples, then what you are calling Hold-out validation is really just 2-fold ($k$ = 2) cross-validation. Cross-validation is merely a tool for estimating the out-of-sample error rates (or generalizability) of a particular model. The need to estimate the out-of-sample error rate is a common one and has spawned an entire literature. See, for starters, chapter 7 of ESL.

So to answer the questions:

  1. Why talk about it? Pedagogically. It's worthwhile to think of Hold-out validation as a special - and only occasionally useful - case of an otherwise quite useful method with many, many variations.

  2. Why use it? If one is lucky enough to have a colossal dataset (in terms of observations, $n$), then splitting the data in half - training on one half and testing on the other - makes sense. This makes sense for computational reasons since all that is required is fitting once and predicting once (rather than $k$ times). And it makes sense from a "large-sample estimation" perspective since you have a ton of observations to fit your model to.

A rule-of-thumb I've learned is: when $n$ is large, $k$ can be small, but when $n$ is small, $k$ should be close to $n$.

share|improve this answer
I don't think that holdout is the same as 2 fold validation, because in 2 fold validation you will fit two models and then average out the errors across the two holdout sets. – Alex Aug 31 '15 at 0:30

If your model selection & fitting procedure can't be coded up because it's subjective, or partly so,—involving looking at graphs & the like—hold-out validation might be the best you can do. (I suppose you could perhaps use something like Mechanical Turk in each CV fold, though I've never heard of its being done.)

share|improve this answer

All these are useful comments. Just take one more into account. When you have enough data, using Hold-Out is a way to assess a specific model (a specific SVM model, a specific CART model, etc), whereas if you use other cross-validation procedures you are assessing methodologies (under your problem conditions) rather than models (SVM methodology, CART methodology, etc).

Hope this is helpful!

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.