0
$\begingroup$

I'm trying to derive the variance of the slope parameter for a simple linear regression in the following way, however I'm running into an issue I don't know how to resolve. Define $y_i=\beta_0+\beta_1\cdot x_i+\epsilon_i$ where $\epsilon_i\sim N(0,\sigma^2)$. Let $\hat\beta_0$ and $\hat\beta_1$ be the least-square estimators for the intercept and slope parameters. My derivation is as follows:

$ \begin{gather*} \\ Var(\hat\beta_1) = Var\left(\frac{\sum(x_i-\overline{x})(y_i-\overline{y})}{\sum(x_i-\overline{x})^2} \right) \\ \text{* since x's are fixed, they act as constants in this derivation } \\ = \frac{1}{(\sum(x_i-\overline{x})^2)^2}\sum(x_i-\overline{x})^2Var(y_i-\overline{y}) \\ = \frac{1}{(\sum(x_i-\overline{x})^2)^2}\sum(x_i-\overline{x})^2[Var(y_i)-Var(\overline{y})] \\ = \frac{1}{(\sum(x_i-\overline{x})^2)^2}\sum(x_i-\overline{x})^2\left[\sigma^2-\frac{\sigma^2}{n}\right] \\ = \frac{1}{(\sum(x_i-\overline{x})^2)^2}\left[\sigma^2-\frac{\sigma^2}{n}\right]\cdot\sum(x_i-\overline{x})^2 \\ = \frac{\left[\sigma^2-\frac{\sigma^2}{n}\right]}{\sum(x_i-\overline{x})}\ne\frac{\sigma^2}{\sum(x_i-\overline{x})} \end{gather*} $

My only thought is that I'm forgetting to account for the covariance between $y_i$ and $\overline{y}$, but I thought that this was simply 0. Any insight would be much appreciated!

Edit : I know there is a way with defining $\hat\beta_1$ as the linear combination of some constants, but I was curious if there is a simple fix to my steps

Edit 2 : I thought the covariance is accounted for by the following:

$ \begin{gather*} Var(y_i-\overline{y}) =Var\left(y_i-\frac{\sum{y_i}}{n}\right)\\ =Var\left(y_i-\frac{y_1}{n}-\frac{y_2}{n}-\ldots-\frac{y_i}{n}-\ldots-\frac{y_n}{n}\right)\\ =Var\left(\left(1-\frac{1}{n}\right)\cdot y_i-\frac{y_1}{n}-\ldots-\frac{y_n}{n}\right)\\ =\left(1-\frac{1}{n}\right)^2\cdot Var(y_i)+\frac{1}{n^2}\cdot Var(y_1)+\ldots+\frac{1}{n^2}\cdot Var(y_n)-\frac{2}{n^2}\mathop{\sum{}\sum{}}_{j\ne k}Cov(y_j,y_k)\\ \text{**I'm probably missing some rigour with the notation of the double sum, but you get the point}\\ \text{The covariances between different response values are equal to 0 by definition (since $\epsilon_i$ are i.i.d). Thus:}\\ =\left(\frac{n-1}{n}\right)^2\cdot \sigma^2+\frac{n-1}{n^2}\sigma^2\\ =\frac{\sigma^2}{n^2}((n-1)^2+(n-1))\\ =\frac{\sigma^2}{n^2}((n-1)(n-1+1)\\ =\frac{(n-1)\sigma^2}{n} \end{gather*} $

$\endgroup$
5
  • $\begingroup$ You can get to the core idea by using a more abstract notation: write $\hat\beta_1$ as a linear combination of the $y_i$ and go on from there. This helps avoid typographical and algebraic errors, too. $\endgroup$
    – whuber
    Commented Oct 5 at 12:25
  • $\begingroup$ @whuber, the problem is OP doesn't want that. (That's why I didn't vtc the otherwise countless posts as possible dupe). They specifically want to know why the stage at question was flawed. $\endgroup$ Commented Oct 5 at 12:38
  • $\begingroup$ But that doesn't mean I disagree with the closure. That's the standard procedure that circumvents any algebraic, as well as any independence issues. $\endgroup$ Commented Oct 5 at 12:40
  • $\begingroup$ @User1865345 The issue for me is that the question in its current form is only about algebra. All the statistical issues have been addressed in other threads, such as the one I nominated as the duplicate. $\endgroup$
    – whuber
    Commented Oct 5 at 15:04
  • $\begingroup$ I thought this stack exchange would also be able to help me with insight on an algebraic mis-step that makes my derivation wrong. I know there are ways to circumvent the issue with the covariances (as I mentioned in the post), but I was curious about where my steps are falling short. I don't understand how the suggested answer has anything to do with this $\endgroup$
    – aort01
    Commented Oct 5 at 17:10

1 Answer 1

0
$\begingroup$

You could have made the calculations easier by noting that

\begin{align}\frac{\sum (x_i-\bar x) (y_i-\bar y) }{\sum (x_i-\bar x) ^2}&=\frac{\sum(x_i-\bar x) y_i}{\sum(x_i-\bar x) ^2}-\frac{\bar y\sum (x_i-\bar x) }{\sum(x_i-\bar x) ^2},\end{align} where the second term is zero as $\sum (x_i-\bar x) =0.$ Then you could have expressed $\hat \beta_1 =\sum c_i y_i$ where $c_i:=(x_i-\bar x) /\sum(x_i-\bar x) ^2.$ As $\sum c_i^2 =1/\sum(x_i-\bar x) ^2, $ the variance of $\hat \beta_1$ is $\sigma^2/\sum(x_i-\bar x) ^2.$

Also, a remainder that $\operatorname{Var}(X\pm Y) =\operatorname{Var}(X) + \operatorname{Var}(Y) \pm 2\operatorname{Cov}(X,Y);$ check whether you used that correctly in your calculations. And is $y$ and $\bar y$ independent?

$\endgroup$
3
  • $\begingroup$ I added an edit to my original post. While I agree with you that using the linear combinations is easier, I dont understand why my derivation leaves me to something incorrect. I've included my reasonings for an exclusion of the covariance, it is possible my reasonings are flawed $\endgroup$
    – aort01
    Commented Oct 5 at 4:39
  • $\begingroup$ Are $y_i-\bar y$s independent? In your first calculation, you somehow operated the $\operatorname{Var}$ by taking $(x_i-\bar x) $ out (without squaring it) and worked with separate terms of $\operatorname{Var}(y_i-\bar y) ;$ is it justifiable? Again, are the variables $y_i-\bar y$ independent? $\endgroup$ Commented Oct 5 at 4:54
  • $\begingroup$ You are correct, taking out the $(x_i-\overline{x})$ without squaring was a typo. I'll have to do some more digging on the independence of $y_i$ and $\overline{y}$, there must be something i'm overlooking. I appreciate the guidance! $\endgroup$
    – aort01
    Commented Oct 5 at 4:58

Not the answer you're looking for? Browse other questions tagged or ask your own question.