Cross validation (CV) seems to be a simple and useful tool for estimating the out-of-sample error of a linear regression model, even though it is rarely used for this purpose. Why that? Is there a better ways for error estimation? Is it possible to estimate the out-of-sample error by only using the training data?
$\begingroup$
$\endgroup$
-
3$\begingroup$ You need to state your ultimate goal. For linear models we have a closed-form expression for an unbiased estimate of residual variance if you are honest about the predictor degrees of freedom in its denominator. $\endgroup$ – Frank Harrell Nov 3 '14 at 16:38
-
1$\begingroup$ "honest about the predictor degrees of freedom" ... which can become difficult if the data has (or possibly has) a clustered structure. And yes, depending on what you want/need to measure, other procedures may be more appropriate. $\endgroup$ – cbeleites unhappy with SX Nov 3 '14 at 16:45
-
2$\begingroup$ Perhaps, but a bit more important is that all the predictors were pre-specified and none was deleted by seeming to be 'unimportant'. Otherwise, d.f. needs to be the 'effective d.f.' as in citeulike.org/user/harrelfe/article/13265069 $\endgroup$ – Frank Harrell Nov 3 '14 at 17:46
-
$\begingroup$ Given the assumptions that go into the closed-form expression and the potential to run into the multiple comparisons problem before you even choose what to regress on, is it not usually better to rely on cross-validation and a test set to estimate the predictive power of a linear model? $\endgroup$ – rinspy Jul 31 '17 at 9:48
$\begingroup$
$\endgroup$
I see it used quite often in fact. Note that the popular predicted residual error sum of squares (PRESS) statistic calculated from the residuals & diagonal elements of the hat matrix is what you'd get by performing leave-one-out cross validation.
Alternatives to cross-validation include splitting the sample into training & test sets, & bootstrapping to estimate the optimism in any measure of fit & then correcting for it. See Steyerberg et al. (2001), "Internal validation of predictive models: efficiency of some procedures for logistic regression analysis", Journal of Clinical Epidemiology, 54, pp 774 – 781 for a comparison. Splitting the sample isn't a good idea unless you have a large sample size (in the tens of thousands, I've heard).
answered Nov 3 '14 at 12:49
Scortchi - Reinstate Monica♦Scortchi - Reinstate Monica
25.8k7 gold badges69 silver badges221 bronze badges
-
2$\begingroup$ In analytial chemistry it is used so often that the measures get their name accordingly: e.g. $RMSE_{CV}$ $\endgroup$ – cbeleites unhappy with SX Nov 3 '14 at 16:42
