Estimating prediction error

Would appreciate any answer on characterizing/estimating prediction error on future data for nonlinear regression problem. Under what conditions would cross validation error or simple test error on randomly selected 20% of data available be useful to characterize prediction error on new data (expected value, or max/min)? I've heard somewhere that cross validation error is an optimistic estimate, what would be a pessimistic (but somewhat tight upper bound) on prediction error?

-

If you have done cross-validation very carefully (there are many ways to make mistakes that can lead to overly optimistic results) then if your new data is drawn from the same population as the training data, the cross-validation result should be about right. In technical terms cross-validation should return an unbiased estimate of the error, so even if though the test result may vary from expectations, it should be just as likely to be better as it is to be worse.

For a good guide to cross-validation, see chapter 7 of Elements of Statistical Learning. A common mistake in cross-validation is to ensure that any choices you make developing the model such as tuning parameters, deciding which variables are useful and even what algorithm to use, needs to be evaluated via cross-validation.

However, the key assumption is that the test set is from the same population as the training set. In many real world applications of statistical models, the system being modelled is likely to change over time, even if it is in subtle ways such as changes in the ways samples are taken. Any change will degrade the performance of the model. For this reason, in practical terms cross-validation error on the static training set might be optimistic compared with how some system might perform in the real world. The details will depend entirely on the nature of the data, so there is no single quantitative answer to your question.

-
would you mind pointing me to a resource on how to do cross-validation correctly? –  oalah Jul 17 '12 at 2:53
i shuffled data points, then divided them into 6 non overlapping subsets, taking each in turn as the validation set and the rest as the training. in each round a model is trained by minimizing a regularized RMSE (with L0 norm), the approximation error (RMSE) on the validation set is taken, and after completing 6 rounds of train-validate, the validation errors are averaged. Just in case, i did 10 repeats of the shuffle, and 6-fold cross validation. The avg CV errors are quite similar from repeat to repeat, except for two repeats, where one validation set produced comparatively very high RMSE. –  oalah Jul 17 '12 at 3:01
I added a good reference for doing CV the right and the wrong way. From the description you have just given I can't see anything you are doing wrong. –  Bogdanovist Jul 17 '12 at 3:05
thanks @Bogdanovist –  oalah Jul 18 '12 at 12:20