3
$\begingroup$

Suppose we aim to predict $Y$ from $X$ using the linear regression model $Y = mX + b$. There is a standard variance decomposition:

$$\operatorname{Var}[Y] = \operatorname{Var}[\widehat{Y}] + \operatorname{Var}[R],$$

where $\widehat{Y}=mX+b$ is the model's prediction at $X$ and $R=Y-\widehat{Y}$ is the residual. Thus, the variance of the $Y$'s is equal to the sum of the variance of the predictions and the variance of the residuals.

I can derive this algebraically. But is there a simple graphical proof? Or some other way to see why this is true, in a way that doesn't require much in the way of formulas or algebraic derivations?

Improve this question
$\endgroup$
  • $\begingroup$ In subject space, Prediction and residuals are perpendicular vectors stats.stackexchange.com/a/124892/3277 with variances being their squared lengths. Therefore, according to pythagorean theorem..... $\endgroup$ – ttnphns Nov 13 '18 at 8:59
  • $\begingroup$ @ttnphns, interesting! How can we see that they are perpendicular? In other words, why are they guaranteed to be perpendicular? $\endgroup$ – D.W. Nov 13 '18 at 22:48
0
$\begingroup$

How about simply:

enter image description here

$Var[Y] = Var[red] + Var[blue]$

Code:

x <- rnorm(10)
y <- rnorm(10, x)
plot(x, y, pch=19, col="blue", bty="n")
m <- lm(y ~ x)
abline(m, col="red")
ypred <- predict(m, newdata=data.frame(x=x))
segments(x, ypred, x, y, col="blue")
Improve this answer
$\endgroup$
  • $\begingroup$ Yes, that is a good picture illustrating what that equation is saying. But I'm asking for something to help explain why the equation is true. Why should it be true that Var[y] = Var[red] + Var[blue]? I'm not seeing how to get that from the picture. Did you have some thoughts on that? Apologies if my question was confusing/unclear. $\endgroup$ – D.W. Nov 13 '18 at 22:44
  • $\begingroup$ Well, I'd say that the figure shows that there is some "red" variance which is what our model explains, but to get to the real y values we need to add the "blue" variance. Maybe a better question would be: how to show graphically why (given certain assumptions) is the sum of variances equal to the variance of the sum? $\endgroup$ – January Nov 14 '18 at 7:51
  • 1
    $\begingroup$ Yes, that is exactly what I am asking. $\endgroup$ – D.W. Nov 14 '18 at 16:10
  • $\begingroup$ OK, so maybe you should ask another question, without referring to linear regression. $\endgroup$ – January Nov 14 '18 at 17:58
  • $\begingroup$ This question is specific to linear regression, so it can't be asked without that context. It is not true in general that $\text{Var}[U+V]=\text{Var}[U]+\text{Var}[V]$; it happens to be true in this case, for the specific random variables mentioned in the question, due to some specifics about how those random variables are related, but the question is why -- the question can't be separated from the context. $\endgroup$ – D.W. Nov 14 '18 at 18:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.