Hold-out is often used synonymous with validation with independent test set, although there are crucial differences between splitting the data randomly and designing a validation experiment for independent testing.
Independent test sets can be used to measure generalization performance that cannot be measured by resampling or hold-out validation, e.g. the performance for unknown future cases (= cases that are measured later, after the training is finished). This is important in order to know how long an existing model can be used for new data (think e.g. of instrument drift).
More generally, this may be described as measuring the extrapolation performance in order to define the limits of applicability.
Another scenario where hold-out can actually be beneficial is: it is very easy to ensure that training and test data are properly separated - much easier than for resampling validation: e.g.
- decide splitting (e.g. do random assignment of cases)
- measurement and reference data of the training cases => modeling\
neither measurements nor reference of test cases is handed to the person who models.
- final model + measurements of the held-out cases => prediction
- compare predictions with reference for held-out cases.
Depending on the level of separation you need, each step may be done by someone else. As a first level, not handing over any data (not even the measurements) of the test cases to the modeler allows to be very certain that no test data leaks into the modeling process. At a second level, the final model and test case measurements could be handed over to yet someone else, and so on.
In some fields/cases/applications, we consider this obvious independence sufficiently important to prescribe that an independent organization is needed for validation*, e.g. in clinical chemistry (we also do that e.g. for vehicle safety: the one who safeties your car is not the same as your repair guy, and they are also in separate businesses).
(* I'm chemometrician/analytical chemist. To me, there is not much of a conceptual difference between validating a wet-lab method or an in-silico method (aka predictive model). And the difference will become even less with the advance of machine learning e.g. into medical diagnostics.)
Yes, you pay for that by the lower efficiency of the hold-out estimates compared to resampling validation. But I've seen many papers where I suspect that that the resampling validation does not properly separate cases (in my field we have lots of clustered/hierarchical/grouped data).
I've learned my lesson on data leaks for resampling by retracting a manuscript a week after submission when I found out that I had a previously undetected (by running permutation tests alongside) leak in my splitting procedure (typo in index calculation).
Sometimes hold-out can be more efficient than finding someone who is willing to put in the time to check the resampling code (e.g. for clustered data) in order to gain the same level of certainty about the results.
However, IMHO it is usually not efficient to do this before you are in the stage where you anyways need to measure e.g. future performance (first point) - in other words, when you anyways need to set up a validation experiment for the existing model.
OTOH, in small sample size situations, hold-out is no option: you need to hold out enough test cases so that the test results are precise enough to allow the needed conclusion (remember: 3 correct out of 3 test cases for classification means a binomial 95% confidence interval that ranges well below 50:50 guessing!)
Frank Harrell would point to the rule of thumb that at least ca. 100 (test) cases are needed to properly measure a proportion [such as the fraction of correctly predicted cases] with a useful precision.
Update: there are situations where proper splitting is particularly hard to achieve, and cross validation becomes unfeasible. Consider a problem with a number of confounders. Splitting is easy if these confounders are strictly nested (e.g. a study with a number of patients has several specimen of each patient and analyses a number of cells of each specimen): you split at the highest level of the sampling hierarchy (patient-wise). But you may have independent confounders which are not nested, e.g. day-to-day variation or variance caused by different experimenters running the test. You then need to make sure the split is independent for all confounders on the highest level (the nested confounders will automatically be independent). Taking care of this is very difficult if some confounders are only identified during the study, and designing and performing a validation experiment may be more efficient than dealing with splits that leave almost no data neither for training nor for testing of the surrogate models.