Would appreciate any answer on characterizing/estimating prediction error on future data for nonlinear regression problem. Under what conditions would cross validation error or simple test error on randomly selected 20% of data available be useful to characterize prediction error on new data (expected value, or max/min)? I've heard somewhere that cross validation error is an optimistic estimate, what would be a pessimistic (but somewhat tight upper bound) on prediction error?
$\begingroup$
$\endgroup$
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
3
If you have done cross-validation very carefully (there are many ways to make mistakes that can lead to overly optimistic results) then if your new data is drawn from the same population as the training data, the cross-validation result should be about right. In technical terms cross-validation should return an unbiased estimate of the error, so even if though the test result may vary from expectations, it should be just as likely to be better as it is to be worse.
For a good guide to cross-validation, see chapter 7 of Elements of Statistical Learning. A common mistake in cross-validation is to ensure that any choices you make developing the model such as tuning parameters, deciding which variables are useful and even what algorithm to use, needs to be evaluated via cross-validation.
However, the key assumption is that the test set is from the same population as the training set. In many real world applications of statistical models, the system being modelled is likely to change over time, even if it is in subtle ways such as changes in the ways samples are taken. Any change will degrade the performance of the model. For this reason, in practical terms cross-validation error on the static training set might be optimistic compared with how some system might perform in the real world. The details will depend entirely on the nature of the data, so there is no single quantitative answer to your question.
answered Jul 17, 2012 at 1:59
-
$\begingroup$ would you mind pointing me to a resource on how to do cross-validation correctly? $\endgroup$– oalahJul 17, 2012 at 2:53
-
$\begingroup$ i shuffled data points, then divided them into 6 non overlapping subsets, taking each in turn as the validation set and the rest as the training. in each round a model is trained by minimizing a regularized RMSE (with L0 norm), the approximation error (RMSE) on the validation set is taken, and after completing 6 rounds of train-validate, the validation errors are averaged. Just in case, i did 10 repeats of the shuffle, and 6-fold cross validation. The avg CV errors are quite similar from repeat to repeat, except for two repeats, where one validation set produced comparatively very high RMSE. $\endgroup$– oalahJul 17, 2012 at 3:01
-
1$\begingroup$ I added a good reference for doing CV the right and the wrong way. From the description you have just given I can't see anything you are doing wrong. $\endgroup$ Jul 17, 2012 at 3:05
$\begingroup$
$\endgroup$
0
Let me add to Bogdanovist's excellent answer that
cross validation is unbiased for what it measures: the predicitive abilities of "surrogate" models with respect to the data at hand ("drawn from the same population").
the often stated pessimistic bias arises in situations where the surrogate models are on average worse than the real model, usually because of the smaller training sample size (even if the drawn from the same population assumption is true)
This paper stresses the "drawn from the same population" problems, particularly the drift over time: Esbensen and Geladi: Principles of Proper Validation: use and abuse of re-sampling for validation, Journal of Chemometrics, Volume 24, Issue 3-4, pages 168-187, March-April 2010
answered Jul 17, 2012 at 8:41
cbeleites unhappy with SXcbeleites unhappy with SX
34.7k3 gold badges70 silver badges133 bronze badges