I am interested in how the estimate of the regression $\beta_1$ decreases with respect to sample size $n$. Does this look ok or am I missing something? $$ \begin{align} \text{Var}(\hat \beta_1) & = \frac{\sigma^2}{\sum_{i=1}^n (x_i - \bar x)^2} \\ & \le \frac{\sigma^2}{\sum_{i=1}^n \min_i \{(x_i - \bar x)^2\}} \\ & = \frac{\sigma^2}{n \min_i\{(x_i - \bar x)^2\}} \\ & = O\bigg(\frac{1}{n}\bigg). \end{align} $$

Edit: It seems that the first answer in this post seems to agree with my result. However, the first answer in this post seems to say something else as it says that $\sigma^2$ is itself estimated (by the MSE) and that this estimate has a dependence on $n$.

Can someone clarify exactly how the variance decreases with $n$?

  • 1
    $\begingroup$ The answer depends on your model: exactly what marginal distribution of the explanatory variables are you assuming? This has already been explained in the answers to your previous question at stats.stackexchange.com/questions/491002. $\endgroup$ – whuber Oct 8 '20 at 15:02
  • $\begingroup$ I am not making an assumption on the distribution of the explanatory variables as they are coming from real life observations. $\endgroup$ – ManUtdBloke Oct 8 '20 at 15:16
  • 1
    $\begingroup$ It would seem naturual that the more random observation we have (e.g. the larger $n$ gets), the more the variance is reduced, but I have not seen a formula that makes explicit how the variance decreases as $n$ increases. That is my motivation for asking this question. $\endgroup$ – ManUtdBloke Oct 8 '20 at 15:19
  • $\begingroup$ In kernel density estimation, I have seen proofs where the decay variance of the estimate with respect to $n$ is explicit (as the bandwidth goes to zero), so I figured it would also be possible to have similar explicit results for the decay of the variance of the coefficients in linear regression? Page 7 of faculty.washington.edu/yenchic/18W_425/Lec6_hist_KDE.pdf shows that the variance of a KDE estimate decays as $\frac{1}{nh}p(x_0) \sigma_K^2$, for example. $\endgroup$ – ManUtdBloke Oct 8 '20 at 15:21

The true variance of $\hat\beta_1$ is $\mathcal O(1/n)$, as you correctly proved. The estimated variance of $\beta_1$ is a random variable and its asymptotic behavior is better described using big O probability notation. Since the MSE is a consistent sinais of $\sigma^2$, we have $\hat\sigma^2=\sigma^2+\mathcal o_p(1)$. If we replace $\sigma^2$ by $\hat\sigma^2$ in your proof, we get that the estimated variance of $\beta_1$ is $\mathcal O_p(1/n)$.

TL;DR: the true variance is $\mathcal O(1/n)$, while the estimated variance is $\mathcal O_p(1/n)$ (that is, it is asymptotically bounded by $1/n$ with probability 1).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.