0
$\begingroup$

Which is the best/most reliable representation of a model's predictive accuracy, the accuracy based on the n-fold cross validation or the model's accuracy on an unseen dataset?

At a glance, I would definitely have thought that predicting the unseen dataset would be a better measure of predictive accuracy but on second thoughts it seems likely that this will be smaller than the training dataset and hence more prone to anomalies causing mis-representations in the model's predictive power.

Any references in the answer would be greatly appreciated.

EDIT: Describing current setup - With a dataset of 100k , I randomly selected 10% and removed these building the model on the remaining dataset. I used the 10 % to generate predictions and calculate the accuracy, which of course differs from the accuracy of predictions from cross validation. Which of these realistically represents the model's predictive accuracy more?

Thanks

$\endgroup$
  • 1
    $\begingroup$ Can you clarify the situation you have in mind? What is the setup? People don't usually do both cross validation & have an additional 'unseen dataset'. $\endgroup$ – gung - Reinstate Monica Aug 22 '16 at 22:02
  • $\begingroup$ Added current setup $\endgroup$ – numX Aug 23 '16 at 18:46
3
$\begingroup$

Predictive accuracy always needs to be calculated on unseen data - whether that data is unseen via cross validation splits or via a separate data set.

So often the most important point is to avoid leaks between training and test data. This may be easier to achieve with hold out (e.g. by obtaining test cases only after model training is finished) than for resampling.
But careful: very often "hold out" or "independent test" are used that are in fact a single random split of the available data set. That procedure is of course prone to the same data leaks that cross validation is.

Yes, for simple data, cross validation makes more efficient use of your data. And in small sample size situations, that can be the crucial advantage of resampling. But when you have to deal with multiple confounders and need to split independently for all those confounders, that advantage vanishes very fast because you end up excluding large parts of your data from both test and training set for each surrogate model.

Related:


UPDATE: described scenario of 100k (I assume cases) x unknown no of variates.

That is certainly not a small sample size situation. In this situation, a random hold out set of 10 % = 10000 cases should have no practically relevant difference to cross validation results. The more so, as a random subset is prone to the same data leaks that cross validation is prone to as well: confounders that lead to clustering in the data. If you have such confounders, your effective sample size may be orders of magnitude below the 100k rows, and any kind of splitting that doesn't take care of those confounders will mean a data leak between training and test and lead to overoptimistic bias in the error estimates.

The more efficient use of cases in cross validation is mostly relevant with small data sets where

  1. stability of the model is an issue and must be checked (which is easily done by cross validation), and
  2. uncertainty of the test result due to small numbers of test cases is large
    here cross validation is better as a full run will test each case.

For theory, I recommend reading up the relevant parts of The Elements of Statistical Learning.

These papers have empirical results on bias and variance of different validation schemes (though they deal explicitly with small sample size situations):

$\endgroup$
  • $\begingroup$ Thanks for your reply! The questions referenced seem to suggest that using cross validation is statistically better than hold out, but none of them really quote any formal references stating why, do you happen to have any ? $\endgroup$ – numX Aug 23 '16 at 18:57
1
$\begingroup$

I think it's pretty plain evident that cross validation is statistically better for estimating parameters or checking stability. That's because it measures prediction error ideally over all data with equal weights. But that brings you to statistics of course. Because traditional prediction errors and model fit statistics do the job as well, if you have the right model. So in principle, if you believe in your model, you don't need to do prediction at all, you just fit your model by using all data and calculate prediction errors from statistical theory. That's what most scientists and scholars do, I believe.

But if you believe that you don't have the right model, and you are just using a wrong model to mimic the data, then your question becomes relevant. As said, I would use cross validation to calibrate the parameters, but to demonstrate predictive ability, I would use a separate validation data set. Why? Because it's more convincing, more understanadable, and finally, because you very likely report the parameters calibrated from the whole data, not from any of you training subsets.

Moreover, I would not just randomly pick the validation data, but I would choose it somehow conceptually different from the training data: For example, past - future, Americans - Europeans. I think that's the ultimate test of model validity. Traditional cross validation (and randomly picked validation data) just measure internal validity (terminology my own). Systematically picked validation data measure external validity which is generalizability. And science is about generalizing patterns.

And needless to say, the prediction gains should be statistically significant. Otherwise, it's just optimistic reporting of noise.

Of course this is a very rigorous take on model validation, and I believe most modelers actually fail one or two: cross validation, external validation or statistical significance. Because it's all too easy to overfit a model in small data and then go like 'whoah, what good predictions'. But I've also seen examples to the contrary: In cancer research, it was every day practice to first fit a random forest by cross validation, and then to demonstrate predictive ability on different patients. Because in that way they could make the doctors and biologists to believe in the result.

This is my take on the issue as a practicing statistician some 5+ years into the working life.

$\endgroup$
  • 1
    $\begingroup$ This answer could be improved by adding some references to support your assertions. $\endgroup$ – Silverfish Aug 25 '16 at 10:40
  • $\begingroup$ Thanks, could you add some references as indicated by Silverfish please? $\endgroup$ – numX Aug 25 '16 at 17:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.