5
$\begingroup$

In most of the IS (Information Systems) papers I read, the authors do not use a global test set, but only cross-validation and present these results. They reason this makes sense because they have a very small n, e.g. in Text Classification (n=400,p=3000). Is this approach correct?

This question was partly asked already here, but less specific: Do we need a test set when using k-fold cross-validation?

$\endgroup$
2
  • $\begingroup$ Are they using the same cross validation loop to both choose algorithm hyperparameters and report an estimate of hold out error? If that's the case, then yes, that is mistaken. $\endgroup$ Commented Feb 13, 2017 at 15:06
  • $\begingroup$ Exactly. While they do not necessarily tune hyper parameters, they use in every case the statistical performance results on the validation set to choose a model with standard parameters (e.g. SVM vs Random Forest vs ANN) which will then be deployed within a predictive information system. $\endgroup$
    – NKu
    Commented Feb 13, 2017 at 15:52

2 Answers 2

5
$\begingroup$

There are 2 separate issues here:

  • "single" vs. nested validation, and
  • cross validation vs. hold out test set

Single vs. nested

The point is that you have to decide what to use the validation results for:

  • either model selection (including hyperparameter tuning, deciding training algorithm, etc.)
  • (x)or as generalization error estimate

So regardless of sample size, if data-driven steps are applied for deciding the final model, generalization error can only be estimated on a data set that was not involved at all in these decision and training procedures. In other words, a nested validation is needed.

Hold out vs. cross validation

If sample size is too small for single (hold out) splits, both levels of validation can be done by cross validation (aka nested cross validation).

do not use a global test set, but only cross-validation

Note that the hold-out test sets I've seen in practice are typically generated by the same random splitting as cross validation sets. So both hold out and cross validation have the same risk of possible confounders violating independence.
Thus, hold out tests are not per se superior than cross validation.

Once you go ahead and plan a validation study covering all the possible confounders you can think of, things look different: it is then typically more efficient (in terms of total no of cases to collect) to obtain a single well-defined independent test set.

$\endgroup$
6
$\begingroup$

I'd just like to add to @cbeleites' answer, which is good, with some quotes from a recent blog post of Frank Harrell's on split-sample validation:

[...] data splitting is an unstable method for validating models or classifiers, especially when the number of subjects is less than about 20,000 (fewer if signal:noise ratio is high). This is because were you to split the data again, develop a new model on the training sample, and test it on the holdout sample, the results are likely to vary significantly.

And

Data are too precious to not be used in model development/parameter estimation. Resampling methods allow the data to be used for both development and validation, and they do a good job in estimating the likely future performance of a model. Data splitting only has an advantage when the test sample is held by another researcher to ensure that the validation is unbiased.

In other words, in most cases you should prefer a resampling-based validation approach, and especially so when you have few observations. In a $n << p$ situation, the modelling process can have super high variance even if you use high-bias methods such penalized regression. If you don't do resampling you might spend a lot of time trying to interpret noise (speaking from experience).


Edit: so what's resampling-based validation anyway?

There are a few ways to do validation by resampling, the common idea is to make new samples from your original sample. A popular and by now quite well-understood method is to use the bootstrap, which would go like this:

  • Make a bootstrap sample by drawing $n$ observations from your original $n$-sized sample with replacement
  • Since you drew the bootstrap sample with replacement, some observations will appear more than once, others won't be in there at all. Roughly about 60% of the observations will be in there, the other 40% won't.
  • Fit your model to the bootstrap sample
  • Estimate your validation statistic by predicting on the observations that aren't in the bootstrap sample
  • Repeat for as long as you have the patience to

This gives you an empirical distribution over the validation statistic which you can use to say something about how well your method will generalize. This could be for eg to estimate uncertainty or to calculate a point estimate (preferably both!).

Other possible methods: repeated holdout set, repeated cross-validation.

$\endgroup$
2
  • $\begingroup$ Thank you for your answer. Please specify "resampling-based validation" $\endgroup$
    – NKu
    Commented Mar 7, 2017 at 16:49
  • $\begingroup$ @NKu I have added a brief explanation to my answer above $\endgroup$
    – einar
    Commented Mar 7, 2017 at 17:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.