23
$\begingroup$

I have found possibly conflicting definitions for the cross validation (CV) statistic and for the generalized cross validation (GCV) statistic associated with a linear model $Y = X\boldsymbol\beta + \boldsymbol\varepsilon$ (with a normal, homoscedastic error vector $\boldsymbol\varepsilon$).

On the one hand, Golub, Heath & Wahba define the GCV estimate $\hat{\lambda}$ as (p. 216)

the minimizer of $V\left(\lambda\right)$ given by $$ V\left(\lambda\right) = \frac{\frac{1}{n} \left\|\left(I - A\left(\lambda\right)\right)y\right\|^2}{\left(\frac{1}{n} \mathrm{tr}\left(I - A\left(\lambda\right)\right)\right)^2} $$ where $A\left(\lambda\right) = X\left(X^T X + n\lambda I\right)^{-1} X^T$

On the other hand, Efron defines the same concept as $V\left(0\right)$ (p. 24), yet he attributes the introduction of this concept to Craven & Wahba, where its definition (p. 377) is essentially the same as Golub, Heath & Wahba's above mentioned definition.

Does this mean that $0$ minimizes $V\left(\lambda\right)$?

Similarly, Golub, Heath & Wahba define the CV estimate of $\lambda$ (p. 217) as the minimizer of

$$ P\left(\lambda\right) = \frac{1}{n}\sum_{k=1}^n \left(\left[X \beta^{(k)}\left(\lambda\right)\right]_k - y_k\right)^2 $$

where $\beta^{\left(k\right)}\left(\lambda\right)$ is the estimate

$$ \hat{\beta}\left(\lambda\right) = \left(X^T X + n \lambda I\right)^{-1} X^T y $$

of $\beta$ with the $k$th data point $y_i$ omitted.

The authors attribute the introduction of the CV estimate (also called the PRESS estimate) to Allen ("Allen's PRESS", ibid.) Yet in Allen's paper, the PRESS estimate is defined (p. 126) as $n P\left(0\right)$ (in Efron's article it is defined as $P\left(0\right)$ (p. 24)).

Again, does this mean that $0$ minimizes $P\left(\lambda\right)$?


  1. Allen, David M. The Relationship Between Variable Selection and Data Agumentation and a Method for Prediction. Technometrics, Vol. 16, No. 1 (February, 1974), pp. 125-127

  2. Craven, Peter and Wahba, Grace. Smoothing Noisy Data with Spline Functions. Numerische Mathematik 31, (1979), pp. 377-403

  3. Efron, Bradley. How Biased Is the Apparent Error Rate of a Logistic Regression? Technical report no. 232. Department of Statistics, Stanford University (April 1985)

  4. Golub, Gene H., Heath and Grace Wahba. Generalized Cross-Validation as a Method for Choosing a Good Ridge Parameter. Technometrics, Vol. 21, No. 2 (May, 1979), pp. 215-223

$\endgroup$
  • 7
    $\begingroup$ Did you forget to mention that this will be fitted with ridge regression and not least squares? I was totally confused about what $\lambda$ was until I saw the paper titles at the bottom $\endgroup$ – shadowtalker May 7 '16 at 14:41
  • 1
    $\begingroup$ Remove Generalized Cross Validation in the title and add Ridge Regression in the title. Here is what GridSearchCV() defaults to for RidgeCV(): $\endgroup$ – HoofarLotusX Sep 27 '18 at 17:28
1
$\begingroup$

I believe the comments are pointing at the answer, but not stating it bluntly. So I'll be blunt.

The V formula cited here is specific to linear ridge regression. They don't say it is the same as PRESS, they say it is a rotation-invariant version of PRESS. The "rotation-invariant" part is what makes this generalized.

Efron's paper is about logistic regression, customized to that context. If you want to see the math translation between the two contexts, the right book to read is Elements of Statistical Learning, 2ed, by Hastie, Tibshirani, and Freedman. They offer that book free, online: https://web.stanford.edu/~hastie/Papers/ESLII.pdf. Another helpful read on GCV is Generalized Additive Models by Simon Wood. His treatment integrates GCV in general with applications in regression and logistic regression.

If you look at the ESL book, p 244, you see basically the same symbology. They refer to that large matrix product you have as the Smoother matrix (I'd say its a Hat matrix, or a near cousin). They describe the Smoother $S$ as the mapping from $y$ to $\hat{y}$

$$ \hat{y}=S y $$

$S$ can be used to calculate leave one out CV values, one for each row in the data. For linear models, the $S$ matrix plays the role of the Hat matrix in regression diagnostics. However, they say it may be computationally challenging or unnecessary to work that out, and the GCV approach is a slightly more general version of same idea.

They offer a formula for the approximation of GCV:

$$ GCV(\hat{f})=\frac{1}{N}\sum_{i=1}^{N}\left[\frac{y_i - \hat{f}(x_i)}{1-trace(S)/N}\right]^2 $$

This is quite similar in behavior to the AIC in many models. The $trace{S}$ is the effective number of parameters.

The $n\lambda$ piece you quote is more generally a trace of $S$. As far as I can understand, in the abstract GCV is an approximate version of leave one out crossvalidation, but in some cases, (I believe ridge regression), it is exact. That's a main point in the Golub paper.

Good luck, write back if you learn more.

$\endgroup$
  • $\begingroup$ Thanks. I've posted my question over 5 years ago, and since then I have forgotten most of this material, so I can't assess your answer to tell if it's good (which it appears to be) or bad, and, for this reason I can't accept it either. Thanks for posting, though. Hopefully it will be useful for others who might come across this page. $\endgroup$ – Evan Aad Jan 22 at 7:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.