# Hold-out Validation vs K-Fold Validation?

To me, it seems that Hold-out validation is useless. That is, splitting the original dataset into two-parts (training and testing) and using the testing score as a generalization measure, is somewhat useless.

K-Fold seems to give better approximations of generalization (as it trains and tests on every point). So, why would we use the standard Hold-out validation? Or even talk about it?

Thanks

-

My only guess is that you can Hold-Out with three hours of programming experience; the other takes a week in principle and six months in practice.

In principle it's simple, but writing code is tedious and time-consuming. As Linus Torvalds famously said, "Bad programmers worry about the code. Good programmers worry about data structures and their relationships." Many of the people doing statistics are bad programmers, through no fault of their own. Doing k-fold cross validation efficiently (and by that I mean, in a way that isn't horribly frustrating to debug and use more than once) in R requires a vague understanding of data structures, but data structures are generally skipped in "intro to statistical programming" tutorials. It's like the old person using the Internet for the first time. It's really not hard, it just takes an extra half hour or so to figure out the first time, but it's brand new and that makes it confusing, so it's easy to ignore.

You have questions like this: How to implement a hold-out validation in R. No offense intended, whatsoever, to the asker. But many people just are not code-literate. The fact that people are doing cross-validation at all is enough to make me happy.

It sounds silly and trivial, but this comes from personal experience, having been that guy and having worked with many people who were that guy.

-
Maybe as someone who has majored in CS I have a slightly skewed view on this, but if you can implement hold-out validation correctly (which already means splitting the dataset into 2 parts and using one for training and the other for testing), the only thing you need to change is the ratio of the split and put the whole thing into a loop. It just seems hard to believe that this would be a big problem. –  Voo Jun 25 '14 at 21:15
@Voo: in addition, being able to program is not enough here: you must understand the problem well enough to be able to judge for which confounders you need to account during your splitting procedure. See e.g. stats.stackexchange.com/questions/20010/…. I think I see this kind of problems more often than "pure" coding problems (although one never knows: someone who's barely able to code a plain splitting of the rows in the data matrix will usually also make the higher-level mistake of not splitting e.g. at patient level) –  cbeleites Jun 26 '14 at 12:21
Note also that you can do proper (e.g. patient/measurement day/...) hold-out splitting without any programming at all by separating the files the measurement instrument produces... –  cbeleites Jun 26 '14 at 12:23
To the up-voters: note that I asked a separate question that questions my logic. stats.stackexchange.com/q/108345/36229 –  ssdecontrol Jul 20 '14 at 13:29

Hold-out is often used synonymous with validation with independent test set, although there are crucial differences between splitting the data randomly and designing a validation experiment for independent testing.

Independent test sets can be used to measure generalization performance that cannot be measured by resampling or hold-out validation, e.g. the performance for unknown future cases (= cases that are measured later, after the training is finished). This is important in order to know how long an existing model can be used for new data (think e.g. of instrument drift). More generally, this may be described as measuring the extrapolation performance in order to define the limits of applicability.

Another scenario where hold-out can actually be beneficial is: it is very easy to ensure that training and test data are properly separated - much easier than for resampling validation: e.g.

1. decide splitting (e.g. do random assignment of cases)
2. measure
3. measurement and reference data of the training cases => modeling\ neither measurements nor reference of test cases is handed to the person who models.
4. final model + measurements of the held-out cases => prediction
5. compare predictions with reference for held-out cases.

Depending on the level of separation you need, each step may be done by someone else. As a first level, not handing over any data (not even the measurements) of the test cases to the modeler allows to be very certain that no test data leaks into the modeling process. At a second level, the final model and test case measurements could be handed over to yet someone else, and so on.

Yes, you pay for that by the lower efficiency of the hold-out estimates compared to resampling validation. But I've seen many papers where I suspect that that the resampling validation does not properly separate cases (in my field we have lots of clustered/hierarchical/grouped data).

I've learned my lesson on data leaks for resampling by retracting a manuscript a week after submission when I found out that I had a previously undetected (by running permutation tests alongside) leak in my splitting procedure (typo in index calculation).

Sometimes hold-out can be more efficient than finding someone who is willing to put in the time to check the resampling code (e.g. for clustered data) in order to gain the same level of certainty about the results. However, IMHO it is usually not efficient to do this before you are in the stage where you anyways need to measure e.g. future performance (first point) - in other words, when you anyways need to set up a validation experiment for the existing model.

OTOH, in small sample size situations, hold-out is no option: you need to hold out enough test cases so that the test results are precise enough to allow the needed conclusion (remember: 3 correct out of 3 test cases for classification means a binomial 95% confidence interval that ranges well below 50:50 guessing!) Frank Harrell would point to the rule of thumb that at least ca. 100 (test) cases are needed to properly measure a proportion [such as the fraction of correctly predicted cases] with a useful precision.

-
I wish I could give more than +1 for this very thorough answer. I particularly liked you mentioning your issue with a data leak as it effectively illustrates that it can be far from trivial to rule out such problems, even for experts. This is a good reality check! –  Marc Claesen Jun 26 '14 at 11:17

It might be useful to clear up the terminology a little. If we let $k$ be some integer less than (or equal to) $n$ where $n$ is the sample size and we partition the sample into $k$ unique subsamples, then what you are calling Hold-out validation is really just 2-fold ($k$ = 2) cross-validation. Cross-validation is merely a tool for estimating the out-of-sample error rates (or generalizability) of a particular model. The need to estimate the out-of-sample error rate is a common one and has spawned an entire literature. See, for starters, chapter 7 of ESL.

2. Why use it? If one is lucky enough to have a colossal dataset (in terms of observations, $n$), then splitting the data in half - training on one half and testing on the other - makes sense. This makes sense for computational reasons since all that is required is fitting once and predicting once (rather than $k$ times). And it makes sense from a "large-sample estimation" perspective since you have a ton of observations to fit your model to.
A rule-of-thumb I've learned is: when $n$ is large, $k$ can be small, but when $n$ is small, $k$ should be close to $n$.