In most of the IS (Information Systems) papers I read, the authors do not use a global test set, but only cross-validation and present these results. They reason this makes sense because they have a very small n, e.g. in Text Classification (n=400,p=3000). Is this approach correct?
This question was partly asked already here, but less specific: Do we need a test set when using k-fold cross-validation?
There are 2 separate issues here:
The point is that you have to decide what to use the validation results for:
So regardless of sample size, if data-driven steps are applied for deciding the final model, generalization error can only be estimated on a data set that was not involved at all in these decision and training procedures. In other words, a nested validation is needed.
If sample size is too small for single (hold out) splits, both levels of validation can be done by cross validation (aka nested cross validation).
do not use a global test set, but only cross-validation
Note that the hold-out test sets I've seen in practice are typically generated by the same random splitting as cross validation sets. So both hold out and cross validation have the same risk of possible confounders violating independence.
Thus, hold out tests are not per se superior than cross validation.
Once you go ahead and plan a validation study covering all the possible confounders you can think of, things look different: it is then typically more efficient (in terms of total no of cases to collect) to obtain a single well-defined independent test set.
I'd just like to add to @cbeleites' answer, which is good, with some quotes from a recent blog post of Frank Harrell's on split-sample validation:
[...] data splitting is an unstable method for validating models or classifiers, especially when the number of subjects is less than about 20,000 (fewer if signal:noise ratio is high). This is because were you to split the data again, develop a new model on the training sample, and test it on the holdout sample, the results are likely to vary significantly.
And
Data are too precious to not be used in model development/parameter estimation. Resampling methods allow the data to be used for both development and validation, and they do a good job in estimating the likely future performance of a model. Data splitting only has an advantage when the test sample is held by another researcher to ensure that the validation is unbiased.
In other words, in most cases you should prefer a resampling-based validation approach, and especially so when you have few observations. In a $n << p$ situation, the modelling process can have super high variance even if you use high-bias methods such penalized regression. If you don't do resampling you might spend a lot of time trying to interpret noise (speaking from experience).
Edit: so what's resampling-based validation anyway?
There are a few ways to do validation by resampling, the common idea is to make new samples from your original sample. A popular and by now quite well-understood method is to use the bootstrap, which would go like this:
This gives you an empirical distribution over the validation statistic which you can use to say something about how well your method will generalize. This could be for eg to estimate uncertainty or to calculate a point estimate (preferably both!).
Other possible methods: repeated holdout set, repeated cross-validation.
