Expected Prediction Error in a linear model is defined as follows:

$E[\sum_i (y^*_i-\hat{y_i})^2] = E[\sum_i (y_i-\hat{y_i})^2] + 2\sum_i Cov(y_i,\hat{y_i})$

where $y_i$ are the training values, and $y^*_i$ are the test values (Proof Efron (1986)).

Now if we consider a linear regression setting, based on the above equality, my question is instead of doing cross validation why don't we simply minimize this expression?

The first term on the right hand side can be approximated by Residual Sum of Squares. The problem is to estimate the second term.

  • $\begingroup$ Efron had a number of publications in 1986... $\endgroup$ – Richard Hardy May 7 '17 at 8:39

This is possible to do. It's called Mallow's Cp. This criteria gives an unbiased estimate of the test set error of data at data with the same input as the training data. (i.e. the $X$'s are the same, but the $y$'s are different.)

However, we know that, in general, unbiased estimation isn't always the best (due to a bias/variance tradeoff.) It's possible that there are much better estimates of the test error out there. A great reference for better estimated of the test error is here.

Another pro of cross validation is that it also measures the test error at not-necessarily the same inputs as that which were used in the training data set which fit the model

  • $\begingroup$ Mallow's Cp assumes knowledge of the error variance, or estimate it from the larger parameter set, which will severly underestimate it. Moreover it requires the estimation procedure to be OLS. I am asking how to estimate the covariance term on the right directly from data and have a combined minimization procedure. The procedure doesn't necessarily need to be linear. $\endgroup$ – Cowboy Trader May 7 '17 at 6:09
  • $\begingroup$ It seems like if you're interested in minimizing the sum of squared out of sample prediction error, you're already in an OLS setting. In Mallow's Cp (and SURE (en.wikipedia.org/wiki/Stein%27s_unbiased_risk_estimate) more generally), the estimate for the cov comes from Stein's lemma (en.wikipedia.org/wiki/Stein%27s_lemma) which assumes that the true error is gaussian. Is this not what you want? I suppose there's other ways to estimate this covariance term, but I should point out its amazing that it's possible to estimate it without any parameters in this way to begin with $\endgroup$ – user795305 May 7 '17 at 13:59
  • $\begingroup$ (note also that the focus of that paper I linked above is estimating that covariance in ways that dominate the estimate the SURE uses) $\endgroup$ – user795305 May 7 '17 at 14:06
  • $\begingroup$ As far as I know SURE also requires that we know the variance. I can assume that the noise is gaussian, or if it will help (X,Y) can be multivariate gaussian. The paper was a little too technical. Can you summarize the methodology in a few words? OLS is a linear estimation method. I can use a nonlinear one if required. $\endgroup$ – Cowboy Trader May 7 '17 at 14:07
  • $\begingroup$ Okay! In a few days I'll have more time to write a more detailed reply. For now, I think I understand your question better. For instance, you're okay with using a method nonlinear in $y$, like, for instance, lasso. --- Note these cov estimation problems come down to estimating the "degrees of freedom" of the estimate. It became pretty common in the past for papers to publish estimates for the degrees of freedom of their method (see arxiv.org/pdf/0712.0881.pdf , arxiv.org/pdf/1409.5391.pdf , for instance) $\endgroup$ – user795305 May 7 '17 at 14:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.