Machine learning- Calculating predictive accuracy - cross validation vs accuracy on unseen data?

Question

Which is the best/most reliable representation of a model's predictive accuracy, the accuracy based on the n-fold cross validation or the model's accuracy on an unseen dataset?

At a glance, I would definitely have thought that predicting the unseen dataset would be a better measure of predictive accuracy but on second thoughts it seems likely that this will be smaller than the training dataset and hence more prone to anomalies causing mis-representations in the model's predictive power.

Any references in the answer would be greatly appreciated.

EDIT: Describing current setup - With a dataset of 100k , I randomly selected 10% and removed these building the model on the remaining dataset. I used the 10 % to generate predictions and calculate the accuracy, which of course differs from the accuracy of predictions from cross validation. Which of these realistically represents the model's predictive accuracy more?

Thanks

Answer 1

Predictive accuracy always needs to be calculated on unseen data - whether that data is unseen via cross validation splits or via a separate data set.

So often the most important point is to avoid leaks between training and test data. This may be easier to achieve with hold out (e.g. by obtaining test cases only after model training is finished) than for resampling.
But careful: very often "hold out" or "independent test" are used that are in fact a single random split of the available data set. That procedure is of course prone to the same data leaks that cross validation is.

Yes, for simple data, cross validation makes more efficient use of your data. And in small sample size situations, that can be the crucial advantage of resampling. But when you have to deal with multiple confounders and need to split independently for all those confounders, that advantage vanishes very fast because you end up excluding large parts of your data from both test and training set for each surrogate model.

Related:

• Is hold-out validation a better approximation of "getting new data" than k-fold CV?
• Hold out vs. validation experiment

---

UPDATE: described scenario of 100k (I assume cases) x unknown no of variates.

That is certainly not a small sample size situation. In this situation, a random hold out set of 10 % = 10000 cases should have no practically relevant difference to cross validation results. The more so, as a random subset is prone to the same data leaks that cross validation is prone to as well: confounders that lead to clustering in the data. If you have such confounders, your effective sample size may be orders of magnitude below the 100k rows, and any kind of splitting that doesn't take care of those confounders will mean a data leak between training and test and lead to overoptimistic bias in the error estimates.

The more efficient use of cases in cross validation is mostly relevant with small data sets where

1. stability of the model is an issue and must be checked (which is easily done by cross validation), and
2. uncertainty of the test result due to small numbers of test cases is large
   here cross validation is better as a full run will test each case.

---

For theory, I recommend reading up the relevant parts of The Elements of Statistical Learning.

These papers have empirical results on bias and variance of different validation schemes (though they deal explicitly with small sample size situations):

• Kohavi, R.: A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection, Mellish, C. S. (ed.) Artificial Intelligence Proceedings 14$^th$ International Joint Conference, 20 -- 25. August 1995, Montréal, Québec, Canada, Morgan Kaufmann, USA, 1137 - 1145 (1995).
• Beleites, C.; Baumgartner, R.; Bowman, C.; Somorjai, R.; Steiner, G.; Salzer, R. & Sowa, M. G.: Variance reduction in estimating classification error using sparse datasets, Chemom Intell Lab Syst, 79, 91 - 100 (2005).

Answer 2

I think it's pretty plain evident that cross validation is statistically better for estimating parameters or checking stability. That's because it measures prediction error ideally over all data with equal weights. But that brings you to statistics of course. Because traditional prediction errors and model fit statistics do the job as well, if you have the right model. So in principle, if you believe in your model, you don't need to do prediction at all, you just fit your model by using all data and calculate prediction errors from statistical theory. That's what most scientists and scholars do, I believe.

But if you believe that you don't have the right model, and you are just using a wrong model to mimic the data, then your question becomes relevant. As said, I would use cross validation to calibrate the parameters, but to demonstrate predictive ability, I would use a separate validation data set. Why? Because it's more convincing, more understanadable, and finally, because you very likely report the parameters calibrated from the whole data, not from any of you training subsets.

Moreover, I would not just randomly pick the validation data, but I would choose it somehow conceptually different from the training data: For example, past - future, Americans - Europeans. I think that's the ultimate test of model validity. Traditional cross validation (and randomly picked validation data) just measure internal validity (terminology my own). Systematically picked validation data measure external validity which is generalizability. And science is about generalizing patterns.

And needless to say, the prediction gains should be statistically significant. Otherwise, it's just optimistic reporting of noise.

Of course this is a very rigorous take on model validation, and I believe most modelers actually fail one or two: cross validation, external validation or statistical significance. Because it's all too easy to overfit a model in small data and then go like 'whoah, what good predictions'. But I've also seen examples to the contrary: In cancer research, it was every day practice to first fit a random forest by cross validation, and then to demonstrate predictive ability on different patients. Because in that way they could make the doctors and biologists to believe in the result.

This is my take on the issue as a practicing statistician some 5+ years into the working life.