I noticed when calculating the variance of the slope and intercept estimators in simple linear regression that the final formulae do not contain any $y_i$'s. While I understand the calculation, I was wondering if there is a simple intuitive argument for this. I assume that the errors being i.i.d. with a fixed variance regardless of $x$ or $y$ is key, but I think there might be something a bit deeper here that I am missing. It just seems like there must be a calculation free, logical, explanation of this.

\begin{equation} y_i = \theta_0 + \theta_1x_i + \epsilon_i, \end{equation}

\begin{equation} Var{[\hat{\theta_0}]}=\frac{\sigma^2\sum\limits_{i=1}^{n}x_i^2}{n\sum\limits_{i=1}^{n}x_i^2-\left(\sum\limits_{i=1}^{n}x_i\right)^2} \end{equation}

\begin{equation} Var{[\hat{\theta_1}]}=\frac{n\sigma^2}{n\sum\limits_{i=1}^{n}x_i^2-\left(\sum\limits_{i=1}^{n}x_i\right)^2} \end{equation}

  • $\begingroup$ Those formulas on the denominator should normally be avoided, since they're numerically unstable -- with the possible exception of hand calculation (provided you understand when you're losing accuracy using them and take appropriate steps). $\endgroup$ – Glen_b Aug 29 '17 at 9:55
  • $\begingroup$ @Glen_b can you provide a reference, or explanation so I can further explore this point. $\endgroup$ – WetlabStudent Aug 29 '17 at 11:08
  • $\begingroup$ Up to a scale-factor this is just the variance of the $x$'s, on which see Wikipedia's Algorithms for calculating variance: Naive algorithm, particularly the mentions of cancellation and catastrophic cancellation (see also the article on Loss of significance). Fast, stable one-pass algorithms have been around at least since Welford (1962); the issue is addressed in a number of posts here; it's a problem when the mean is very large compared to s.d. $\endgroup$ – Glen_b Aug 30 '17 at 0:47

Well the $y_i$'s are present in a way in your $\sigma^2$: the larger the spread of the $y_i$'s around $\theta_0 + \theta_1 x_i$, the larger the variances of these estimators.

The clue is that these equations are only correct under the model assumption of constant variance $\sigma^2$. Once you set the $x_i$'s and assume constant variances, the $y_i$'s are known to be $Y_i \sim N(E(Y_i|x_i),\sigma^2)$ and thus no longer "affect" the theoretical variances of the estimators.

| cite | improve this answer | |
  • 1
    $\begingroup$ The $y$'s are present in the estimate of $\sigma^2$. $\endgroup$ – Glen_b Aug 29 '17 at 9:53
  • $\begingroup$ @Glen_b's comment is a really important point, the y's are still present in RSS and hence the estimate $\sigma^2$. $\endgroup$ – WetlabStudent Aug 29 '17 at 11:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.