# Does the variance in the estimates of the coefficients for standard linear regression decrease as $O\bigg(\frac{1}{n}\bigg)$?

I am interested in how the estimate of the regression $$\beta_1$$ decreases with respect to sample size $$n$$. Does this look ok or am I missing something? \begin{align} \text{Var}(\hat \beta_1) & = \frac{\sigma^2}{\sum_{i=1}^n (x_i - \bar x)^2} \\ & \le \frac{\sigma^2}{\sum_{i=1}^n \min_i \{(x_i - \bar x)^2\}} \\ & = \frac{\sigma^2}{n \min_i\{(x_i - \bar x)^2\}} \\ & = O\bigg(\frac{1}{n}\bigg). \end{align}

Edit: It seems that the first answer in this post seems to agree with my result. However, the first answer in this post seems to say something else as it says that $$\sigma^2$$ is itself estimated (by the MSE) and that this estimate has a dependence on $$n$$.

Can someone clarify exactly how the variance decreases with $$n$$?

• The answer depends on your model: exactly what marginal distribution of the explanatory variables are you assuming? This has already been explained in the answers to your previous question at stats.stackexchange.com/questions/491002. – whuber Oct 8 '20 at 15:02
• I am not making an assumption on the distribution of the explanatory variables as they are coming from real life observations. – ManUtdBloke Oct 8 '20 at 15:16
• It would seem naturual that the more random observation we have (e.g. the larger $n$ gets), the more the variance is reduced, but I have not seen a formula that makes explicit how the variance decreases as $n$ increases. That is my motivation for asking this question. – ManUtdBloke Oct 8 '20 at 15:19
• In kernel density estimation, I have seen proofs where the decay variance of the estimate with respect to $n$ is explicit (as the bandwidth goes to zero), so I figured it would also be possible to have similar explicit results for the decay of the variance of the coefficients in linear regression? Page 7 of faculty.washington.edu/yenchic/18W_425/Lec6_hist_KDE.pdf shows that the variance of a KDE estimate decays as $\frac{1}{nh}p(x_0) \sigma_K^2$, for example. – ManUtdBloke Oct 8 '20 at 15:21

The true variance of $$\hat\beta_1$$ is $$\mathcal O(1/n)$$, as you correctly proved. The estimated variance of $$\beta_1$$ is a random variable and its asymptotic behavior is better described using big O probability notation. Since the MSE is a consistent sinais of $$\sigma^2$$, we have $$\hat\sigma^2=\sigma^2+\mathcal o_p(1)$$. If we replace $$\sigma^2$$ by $$\hat\sigma^2$$ in your proof, we get that the estimated variance of $$\beta_1$$ is $$\mathcal O_p(1/n)$$.
TL;DR: the true variance is $$\mathcal O(1/n)$$, while the estimated variance is $$\mathcal O_p(1/n)$$ (that is, it is asymptotically bounded by $$1/n$$ with probability 1).