The variance of residuals in a linear regression is given by : $$Var(e_i)=(1-h_{ii})\sigma^2$$ This means that residuals have a lower variance than the error terms, and the variance of residuals is smaller for observations with higher leverage. I see that from the equation above but I can't get an intuitive explanation of these facts. The only explanation I can think of for the 2nd point is : high leverage makes the fitted value $\hat y_i$ close to $y_i$, which means that their difference (= $e_i$) cannot vary a lot. Are there better explanations.

  • 1
    $\begingroup$ The flexibility afforded by fitting a regression function reduces the variability. One way to see this clearly is to imagine extreme cases of overfitting. Another way is to contemplate the decomposition into sums of squares: part of the error variance is manifested in the variance of the estimated regression function. Another way is to work through the simplest cases, such as the model $y_i = \mu+\varepsilon_i,$ which you can relate to well-understood facts about unbiased estimates of variance. It comes down to what kind of explanation you perceive as "better." $\endgroup$
    – whuber
    Nov 3, 2023 at 22:29


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.