4
$\begingroup$

Cross validation (CV) seems to be a simple and useful tool for estimating the out-of-sample error of a linear regression model, even though it is rarely used for this purpose. Why that? Is there a better ways for error estimation? Is it possible to estimate the out-of-sample error by only using the training data?

$\endgroup$
  • 3
    $\begingroup$ You need to state your ultimate goal. For linear models we have a closed-form expression for an unbiased estimate of residual variance if you are honest about the predictor degrees of freedom in its denominator. $\endgroup$ – Frank Harrell Nov 3 '14 at 16:38
  • 1
    $\begingroup$ "honest about the predictor degrees of freedom" ... which can become difficult if the data has (or possibly has) a clustered structure. And yes, depending on what you want/need to measure, other procedures may be more appropriate. $\endgroup$ – cbeleites unhappy with SX Nov 3 '14 at 16:45
  • 2
    $\begingroup$ Perhaps, but a bit more important is that all the predictors were pre-specified and none was deleted by seeming to be 'unimportant'. Otherwise, d.f. needs to be the 'effective d.f.' as in citeulike.org/user/harrelfe/article/13265069 $\endgroup$ – Frank Harrell Nov 3 '14 at 17:46
  • $\begingroup$ Given the assumptions that go into the closed-form expression and the potential to run into the multiple comparisons problem before you even choose what to regress on, is it not usually better to rely on cross-validation and a test set to estimate the predictive power of a linear model? $\endgroup$ – rinspy Jul 31 '17 at 9:48
9
$\begingroup$

I see it used quite often in fact. Note that the popular predicted residual error sum of squares (PRESS) statistic calculated from the residuals & diagonal elements of the hat matrix is what you'd get by performing leave-one-out cross validation.

Alternatives to cross-validation include splitting the sample into training & test sets, & bootstrapping to estimate the optimism in any measure of fit & then correcting for it. See Steyerberg et al. (2001), "Internal validation of predictive models: efficiency of some procedures for logistic regression analysis", Journal of Clinical Epidemiology, 54, pp 774 – 781 for a comparison. Splitting the sample isn't a good idea unless you have a large sample size (in the tens of thousands, I've heard).

| cite | improve this answer | |
$\endgroup$
  • 2
    $\begingroup$ In analytial chemistry it is used so often that the measures get their name accordingly: e.g. $RMSE_{CV}$ $\endgroup$ – cbeleites unhappy with SX Nov 3 '14 at 16:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.