1
$\begingroup$

I'm reading about resampling methods, and specifically leave-one-out cross-validation.

I understood the method, and how to calculate the estimate of the test MSE (Mean squared error):

enter image description here

In the setup of linear regression we have:

enter image description here

$h_i$ is the leverage of the $i$th observation. I want to know the utility of dividing by $1-h_i$. And why not just use the normal formula of MSE?

$\endgroup$
  • $\begingroup$ Can you please provide a reference from where you read this? It will help contextualising your question better. $\endgroup$ – usεr11852 says Reinstate Monic Dec 1 '18 at 11:48
  • $\begingroup$ Of course. I read this in Introduction to statistical learning by Hastie and Tibshirani. $\endgroup$ – Youssef Esseddiq Dec 1 '18 at 11:51
  • $\begingroup$ Thank you. What confuses you about what the authors explain as: "the residuals for high-leverage points are inflated in this formula by exactly the right amount for this equality to hold". Here by residuals we mean the quantity $y_i - \hat{y}_i$. In general, this form of CV is somewhat uncommon. I think it will help you if you view it in conjunction with generalised cross validation (GCV). $\endgroup$ – usεr11852 says Reinstate Monic Dec 1 '18 at 11:59
  • $\begingroup$ Yes, that quote was unclear for me. For observations with high leverage we are dividing by that quantity which makes the ith term of the sum bigger, while low leverage observation get a slight push. It seems like if it is some kind of way to make the model not underestimate the effect of high leverage points via inflating the MSE. Thank you for your comment I'll try to look at the GCV. $\endgroup$ – Youssef Esseddiq Dec 1 '18 at 12:21
  • 2
    $\begingroup$ The variance of $y_i-\hat{y}_i$ is $\sigma^2(1-h_{ii})$ (a number of posts on site discuss that); consequently, the variance of $\frac{y_i-\hat{y}_i}{1-h_{ii}}$ is $\sigma^2$. It's a way of scaling residuals to have equal variance. $\endgroup$ – Glen_b -Reinstate Monica Dec 2 '18 at 9:05
1
$\begingroup$

The $i$th residual is $y_i-\hat{y}_i$, and it has variance $(1-h_{ii})\sigma^2$ (see Leverages and effect of leverage points for this and other algebra used here) so dividing by $\sqrt{1-h_{ii}}$ we get a constant variance: $$ \DeclareMathOperator{\V}{\mathbb{V}} \V \frac{y_i-\hat{y}_i}{\sqrt{1-h_{ii}}} = \frac1{1-h_{ii}} \sigma^2 (1-h_{ii})=\sigma^2 $$ so the variance is indeed constant. So I suspect the squares in the summation should not include also the denominator, or there is missing a square root. This is related to the concept of a studentized residual, see https://en.wikipedia.org/wiki/Studentized_residual.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.