I'm trying to teach myself about cross-validation – I'm new at this – and I'm having a conceptual problem.

I'm reading the book "Classical and Modern Regression" by Myers, chapter four, which is on cross-validation for model selection, specifically, the section on the PRESS statistic. The intuitive definition of the PRESS statistic – which is defined as the sum of the square residuals that result when you drop a data point, recalculate the regression with the data point removed, and then calculate the residual between the new regression and the dropped data point – makes perfect sense to me.

But the PRESS statistic is not usually calculated this way; it's normally calculated by taking advantage of the Sherman–Woodbury–Morrison theorem, which allows you to compute the PRESS residuals in terms of the ordinary residual, which looks like:

$(y_i - \hat{y}_i)/(1 - x_i^t (X^t X)^{-1} x_i) $

Where the numerator is just the residual of the original regression at data point i. But what if the original regression exactly interpolates the data point, so that the numerator is zero? The intuitive definition of the PRESS seems to indicate that the data point gets dropped, and the new regression would be unlikely to still interpolate the point, so the PRESS residual should be nonzero. But the equation given above indicates that it is either exactly zero or, if the denominator is zero, it's undefined.

Clearly I'm missing something. But what?

  • $\begingroup$ If your residual of a point $i$ is zero, removing that point and re-running the regression will not change the line (it will still go through that point). $\endgroup$ – Affine May 21 '14 at 20:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.