To me, it seems that hold-out validation is useless. That is, splitting the original dataset into two-parts (training and testing) and using the testing score as a generalization measure, is somewhat useless.

K-fold cross-validation seems to give better approximations of generalization (as it trains and tests on every point). So, why would we use the standard hold-out validation? Or even talk about it?

  • 8
    why do you think its useless? You can read Elements of Statistical learning theory section 7 for a formal analysis of its pro's and its con's. Statistically speaking, k-fold is better, but using a test set is not necessarily bad. Intuitively, you need to consider that a test set (when used correctly) is indeed a data set that has not been used at all at training. So its definitively useful in some sense to evaluate a model. Also, k-fold is super expensive, so hold out is sort of an "approximation" to what k-fold does (but for someone with low computational power). – Charlie Parker Dec 8 '15 at 19:38
  • Sure. From a theoretical perspective, K-fold is more precise but SLIGHTLY more computationally expensive. The question was: why not ALWAYS do K-fold cross validation? – user46925 Dec 8 '15 at 21:12
  • 2
    I see. I would argue that the reason is mostly always computational. K-fold approximates the generalization error better so from a statistical point of view K-fold is the method of choice I believe. Hold-out is much simpler to implement AND doesn't require training as many models. In practice, training a model can be quite expensive. – Charlie Parker Dec 9 '15 at 19:11
  • 1
    Right - but I think the "too computational expensive" argument is fairly frail. Almost all the time, we're aiming to develop the most accurate models. Yet there is this paradox where a lot of the experiments conducted in the literature only have a single hold-out validation set. – user46925 Dec 9 '15 at 19:55
  • 1
    Question - Elements of Statistical learning theory section 7.10.1 titled "K fold cross validation" seems to indicate that keeping test data entirely separate from training data (as in hold out validation) is ideal, and k- fold validation is just a compromise as data is many a times scarce. I'm still quite new to statistics, could you point out how cross validation is in fact more precise? – numX Aug 24 '16 at 19:17

10 Answers 10

up vote 19 down vote accepted

My only guess is that you can Hold-Out with three hours of programming experience; the other takes a week in principle and six months in practice.

In principle it's simple, but writing code is tedious and time-consuming. As Linus Torvalds famously said, "Bad programmers worry about the code. Good programmers worry about data structures and their relationships." Many of the people doing statistics are bad programmers, through no fault of their own. Doing k-fold cross validation efficiently (and by that I mean, in a way that isn't horribly frustrating to debug and use more than once) in R requires a vague understanding of data structures, but data structures are generally skipped in "intro to statistical programming" tutorials. It's like the old person using the Internet for the first time. It's really not hard, it just takes an extra half hour or so to figure out the first time, but it's brand new and that makes it confusing, so it's easy to ignore.

You have questions like this: How to implement a hold-out validation in R. No offense intended, whatsoever, to the asker. But many people just are not code-literate. The fact that people are doing cross-validation at all is enough to make me happy.

It sounds silly and trivial, but this comes from personal experience, having been that guy and having worked with many people who were that guy.

  • 14
    Maybe as someone who has majored in CS I have a slightly skewed view on this, but if you can implement hold-out validation correctly (which already means splitting the dataset into 2 parts and using one for training and the other for testing), the only thing you need to change is the ratio of the split and put the whole thing into a loop. It just seems hard to believe that this would be a big problem. – Voo Jun 25 '14 at 21:15
  • 3
    @Voo: in addition, being able to program is not enough here: you must understand the problem well enough to be able to judge for which confounders you need to account during your splitting procedure. See e.g. stats.stackexchange.com/questions/20010/…. I think I see this kind of problems more often than "pure" coding problems (although one never knows: someone who's barely able to code a plain splitting of the rows in the data matrix will usually also make the higher-level mistake of not splitting e.g. at patient level) – cbeleites Jun 26 '14 at 12:21
  • Note also that you can do proper (e.g. patient/measurement day/...) hold-out splitting without any programming at all by separating the files the measurement instrument produces... – cbeleites Jun 26 '14 at 12:23
  • 5
    To the up-voters: note that I asked a separate question that questions my logic. stats.stackexchange.com/q/108345/36229 – shadowtalker Jul 20 '14 at 13:29
  • I don't think the answer explaining the difference between two cross validation methods should ever be human time to learn, absurdly biased and not helpful – rgalbo Jul 26 '17 at 13:39

Hold-out is often used synonymous with validation with independent test set, although there are crucial differences between splitting the data randomly and designing a validation experiment for independent testing.

Independent test sets can be used to measure generalization performance that cannot be measured by resampling or hold-out validation, e.g. the performance for unknown future cases (= cases that are measured later, after the training is finished). This is important in order to know how long an existing model can be used for new data (think e.g. of instrument drift). More generally, this may be described as measuring the extrapolation performance in order to define the limits of applicability.

Another scenario where hold-out can actually be beneficial is: it is very easy to ensure that training and test data are properly separated - much easier than for resampling validation: e.g.

  1. decide splitting (e.g. do random assignment of cases)
  2. measure
  3. measurement and reference data of the training cases => modeling\ neither measurements nor reference of test cases is handed to the person who models.
  4. final model + measurements of the held-out cases => prediction
  5. compare predictions with reference for held-out cases.

Depending on the level of separation you need, each step may be done by someone else. As a first level, not handing over any data (not even the measurements) of the test cases to the modeler allows to be very certain that no test data leaks into the modeling process. At a second level, the final model and test case measurements could be handed over to yet someone else, and so on.

Yes, you pay for that by the lower efficiency of the hold-out estimates compared to resampling validation. But I've seen many papers where I suspect that that the resampling validation does not properly separate cases (in my field we have lots of clustered/hierarchical/grouped data).

I've learned my lesson on data leaks for resampling by retracting a manuscript a week after submission when I found out that I had a previously undetected (by running permutation tests alongside) leak in my splitting procedure (typo in index calculation).

Sometimes hold-out can be more efficient than finding someone who is willing to put in the time to check the resampling code (e.g. for clustered data) in order to gain the same level of certainty about the results. However, IMHO it is usually not efficient to do this before you are in the stage where you anyways need to measure e.g. future performance (first point) - in other words, when you anyways need to set up a validation experiment for the existing model.

OTOH, in small sample size situations, hold-out is no option: you need to hold out enough test cases so that the test results are precise enough to allow the needed conclusion (remember: 3 correct out of 3 test cases for classification means a binomial 95% confidence interval that ranges well below 50:50 guessing!) Frank Harrell would point to the rule of thumb that at least ca. 100 (test) cases are needed to properly measure a proportion [such as the fraction of correctly predicted cases] with a useful precision.


Update: there are situations where proper splitting is particularly hard to achieve, and cross validation becomes unfeasible. Consider a problem with a number of confounders. Splitting is easy if these confounders are strictly nested (e.g. a study with a number of patients has several specimen of each patient and analyses a number of cells of each specimen): you split at the highest level of the sampling hierarchy (patient-wise). But you may have independent confounders which are not nested, e.g. day-to-day variation or variance caused by different experimenters running the test. You then need to make sure the split is independent for all confounders on the highest level (the nested confounders will automatically be independent). Taking care of this is very difficult if some confounders are only identified during the study, and designing and performing a validation experiment may be more efficient than dealing with splits that leave almost no data neither for training nor for testing of the surrogate models.

  • 5
    I wish I could give more than +1 for this very thorough answer. I particularly liked you mentioning your issue with a data leak as it effectively illustrates that it can be far from trivial to rule out such problems, even for experts. This is a good reality check! – Marc Claesen Jun 26 '14 at 11:17
  • Aren't you begging the question? Yes, splitting is hard, due to confounders, but it's hard regardless of whether you're doing a single hold-out validation or k-fold cross-validation, isn't it? (Thanks for an insightful answer regardless!) – Nils von Barth Feb 1 '16 at 5:51
  • 1
    @NilsvonBarth: I don't see how my arguments are circular: the OP asks "why [at all] use hold-out validation", and I give a bunch of practical reasons. The statistically most efficient use of a limited number of cases is not always the most important property of the study design. (Though in my experience it often is, due to extremely limited case numbers: I'm far more often advising for repeated/iterated k-fold CV instead of hold-out). For some confounders physical splitting is possible and easy - and a very efficient way to prevent sneak-previews. Who knows whether we'll find that doubly ... – cbeleites Feb 1 '16 at 20:27
  • blinded statistical data analysis may be needed against too many false positive papers at some point? – cbeleites Feb 1 '16 at 20:28
  • 2
    @NilsvonBarth: Careful with hold-out guaranteeing independence: it is easy to implement hold-out in such a way (by physical hold-out of cases, i.e. test specimen are put away and only measured after the model training is finished), but often the term hold-out is used for what is actually far more like a single random split of the data - and then all the possibilities of making mistakes in the splitting can be made with hold-out as well! – cbeleites Feb 4 '16 at 16:08

It might be useful to clear up the terminology a little. If we let $k$ be some integer less than (or equal to) $n$ where $n$ is the sample size and we partition the sample into $k$ unique subsamples, then what you are calling Hold-out validation is really just 2-fold ($k$ = 2) cross-validation. Cross-validation is merely a tool for estimating the out-of-sample error rates (or generalizability) of a particular model. The need to estimate the out-of-sample error rate is a common one and has spawned an entire literature. See, for starters, chapter 7 of ESL.

So to answer the questions:

  1. Why talk about it? Pedagogically. It's worthwhile to think of Hold-out validation as a special - and only occasionally useful - case of an otherwise quite useful method with many, many variations.

  2. Why use it? If one is lucky enough to have a colossal dataset (in terms of observations, $n$), then splitting the data in half - training on one half and testing on the other - makes sense. This makes sense for computational reasons since all that is required is fitting once and predicting once (rather than $k$ times). And it makes sense from a "large-sample estimation" perspective since you have a ton of observations to fit your model to.

A rule-of-thumb I've learned is: when $n$ is large, $k$ can be small, but when $n$ is small, $k$ should be close to $n$.

  • 12
    I don't think that holdout is the same as 2 fold validation, because in 2 fold validation you will fit two models and then average out the errors across the two holdout sets. – Alex Aug 31 '15 at 0:30

If your model selection & fitting procedure can't be coded up because it's subjective, or partly so,—involving looking at graphs & the like—hold-out validation might be the best you can do. (I suppose you could perhaps use something like Mechanical Turk in each CV fold, though I've never heard of its being done.)

Short answer:

I would recommend to always use CV with at least $k=5$ for:

  • complex models
  • final results that have to adhere validity constraints

You might relax this for:

  • training on really large datasets
  • training simple models
  • prototyping when time is an issue

Some of you mentioned, that programming this in R might be an issue. I recommend you to have a look at the "mlr" package. It wraps different packages in a unified interface, also providing really advanced resampling and performance evaluation methods.

Have a look: http://mlr-org.github.io/mlr-tutorial/release/html/resample/ and: http://mlr-org.github.io/mlr-tutorial/release/html/performance/index.htm

Some more explanation - what CV really does is break the bias variance tradeoff:

Now, the problem that both approaches try to solve is to estimate the generalization error, which is conditional on the data that was used to train a model.

Holdout has a problem with bias and variance:

By making the amount of data that we test on smaller, we introduce variance to our estimated generalization error, as the test data might not represent the underlying distribution very well anymore. This itself does not introduce a bias though, as in expectation the estimated performance will be correct.

Making the training set smaller however introduces a pessimistic bias, as again the underlying distribution is not well represented in the data and the model cannot fit the data as well. Making the training set very small introduces variance as well.

As size of training and test set determine each other, this leaves us with a tradeoff: pessimistic bias vs high variance.

$k$-fold Cross validation tackles this problem by keeping the training set large (a fraction of $\frac{k-1}{k}$ of the data is used for training in every iteration) and dealing with the variance of the test error by resampling. After all iterations, we have tested performance on every observation of the dataset with one learner. Obviously, this requires more computation time than simple holdout.

Cross-validating is especially important for more complex (high variance) learners. Those usually are more expensive computationally as well, which can make the whole process quite time intensive.

Just wanted to add some simple guidelines that Andrew Ng mentioned in our CS 229 class at Stanford regarding cross-validation. These are the practices that he follows in his own work.

Let $m$ be the number of samples in your dataset. If $m\le 20$ use Leave-one-out cross validation.

If $20 < m \le 100$ use k-fold cross validation with a relatively large $k \le m$ keeping in mind computational cost.

If $100 < m \le 1,000,000$ use regular k-fold cross validation $(k = 5)$. Or, if there is not enough computational power and $m > 10,000$, use hold-out cross validation.

If $m \ge 1,000,000$ use hold-out cross validation, but if computational power is available you can use k-fold cross validation $(k = 5)$ if you want to squeeze that extra performance out of your model.

All these are useful comments. Just take one more into account. When you have enough data, using Hold-Out is a way to assess a specific model (a specific SVM model, a specific CART model, etc), whereas if you use other cross-validation procedures you are assessing methodologies (under your problem conditions) rather than models (SVM methodology, CART methodology, etc).

Hope this is helpful!

Modeling with time serious data is an exception for me. K fold cannot work in some cases when you need to predict the future based on the previous data. The test sets have to be the future data, and you can never touch them in training phase. e.x predicting sell or the stock market. Hold out is useful in those cases.

Simply put; time. Cross-validation you run the training routine k times (i.e. once for each hold-out set). If you have large data, then you it might take many hours or even days to train the model for just one data set, so you multiply that by k when using cross-validation.

So although cross-validation is the best method, in certain circumstances it's not feasible, and the time it would take might have been better spent modeling the data in different ways, or trying out different loss functions in order to get a better model.

My personal preference is to take validation data from throughout the data set, so rather than take a single 10% chunk from the head or tail of the data, I take 2% from 5 points in the data set. That makes the validation data a bit more representative of the data as a whole.

  • Even though it's an old question and a new answer, I'm voting this up because it challenges the groundless assertion that "K-fold is more precise but SLIGHTLY more computationally expensive", which the other answers were ignoring or passing over too quickly. – The Laconic Jun 24 at 3:12

It should be noted that it's not always possible to apply the cross-validation. Consider the time-dependent datasets such that you want to use the historical data to train a predictive model for the future behaviour. In this case, you have to apply hold-out validation.

  • 2
    In this case, you should do forward validation. – Neil G Sep 8 at 16:54

Your Answer

 
discard

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.