Do I need a global test set when using k-fold cross-validation and a small n (n<In most of the IS (Information Systems) papers I read, the authors do not use a global test set, but only cross-validation and present these results. They reason this makes sense because they have a very small n, e.g. in Text Classification (n=400,p=3000). Is this approach correct?
This question was partly asked already here, but less specific: Do we need a test set when using k-fold cross-validation?
 A: I'd just like to add to @cbeleites' answer, which is good, with some quotes from a recent blog post of Frank Harrell's on split-sample validation:

[...] data splitting is an unstable method for validating models or classifiers, especially when the number of subjects is less than about 20,000 (fewer if signal:noise ratio is high). This is because were you to split the data again, develop a new model on the training sample, and test it on the holdout sample, the results are likely to vary significantly.

And

Data are too precious to not be used in model development/parameter estimation.  Resampling methods allow the data to be used for both development and validation, and they do a good job in estimating the likely future performance of a model. Data splitting only has an advantage when the test sample is held by another researcher to ensure that the validation is unbiased.

In other words, in most cases you should prefer a resampling-based validation approach, and especially so when you have few observations. In a $n << p$ situation, the modelling process can have super high variance even if you use high-bias methods such penalized regression. If you don't do resampling you might spend a lot of time trying to interpret noise (speaking from experience).

Edit: so what's resampling-based validation anyway?
There are a few ways to do validation by resampling, the common idea is to make new samples from your original sample. A popular and by now quite well-understood method is to use the bootstrap, which would go like this:


*

*Make a bootstrap sample by drawing $n$ observations from your original $n$-sized sample with replacement

*Since you drew the bootstrap sample with replacement, some observations will appear more than once, others won't be in there at all. Roughly about 60% of the observations will be in there, the other 40% won't.

*Fit your model to the bootstrap sample

*Estimate your validation statistic by predicting on the observations that aren't in the bootstrap sample

*Repeat for as long as you have the patience to


This gives you an empirical distribution over the validation statistic which you can use to say something about how well your method will generalize. This could be for eg to estimate uncertainty or to calculate a point estimate (preferably both!).
Other possible methods: repeated holdout set, repeated cross-validation.
A: There are 2 separate issues here:


*

*"single" vs. nested validation, and

*cross validation vs. hold out test set


Single vs. nested
The point is that you have to decide what to use the validation results for:


*

*either model selection (including hyperparameter tuning, deciding training algorithm, etc.)

*(x)or as generalization error estimate


So regardless of sample size, if data-driven steps are applied for deciding the final model, generalization error can only be estimated on a data set that was not involved at all in these decision and training procedures. In other words, a nested validation is needed. 
Hold out vs. cross validation
If sample size is too small for single (hold out) splits, both levels of validation can be done by cross validation (aka nested cross validation).

do not use a global test set, but only cross-validation

Note that the hold-out test sets I've seen in practice are typically generated by the same random splitting as cross validation sets. So both hold out and cross validation have the same risk of possible confounders violating independence.
Thus, hold out tests are not per se superior than cross validation.
Once you go ahead and plan a validation study covering all the possible confounders you can think of, things look different: it is then typically more efficient (in terms of total no of cases to collect) to obtain a single well-defined independent test set.
