0
$\begingroup$

I was reviewing the derivation for the variance of ordinary least squares estimators and experienced some confusion.

$$ \Large Var(\hat{\beta}) = \frac{\sigma^2}{\Sigma^n_{i=1}(X_i-\bar{X})^2} $$

From this article, there was this formula, which said in an OLS model where $y_i = \alpha + \beta x_i + \epsilon$, and $\epsilon \sim \mathcal{N}(0,\,\sigma^{2})$, this is a formula that gives the variance of the estimator of $\beta$.

I'm confused what is the sampling distribution that $\hat{\beta}$ comes from? All possible $(x_i, y_i)$ points? Just the $(x_i, y_i)$ points given in the sample dataset? I'm confused on what this variance estimator really means. Any clarity would be helpful.

$\endgroup$

1 Answer 1

1
$\begingroup$

The usual regression model conditions on the $x$'s. They stay the same.

$Y_i$ (not the observed $y_i$ but the random variable that $y_i$ is a realization of) has a distribution that depends on their specific $x_i$. If you and I were both to independently draw a sample with those same $x$'s we'd get different observations, and we'd get different $\hat{\beta}$ values. We'd then each have one draw from the sampling distribution of $\hat{\beta}$, some estimated coefficient $b$. We could call your realized coefficient $b^{[1]}$, and mine $b^{[2]}$ and so forth for any number of them. They're all draws from the conditional distribution (conditional on the x's) of $\hat{\beta}$.

The distribution of $\hat{\beta}$ follows from the joint conditional distribution of the $Y_i$ (the joint distribution conditional on the $x$s).

In the usual model, the vector $\mathbf{Y} = (Y_1,Y_2,...,Y_n)^\top\sim N_n(\mu,\Sigma)$ where $\mu=(\alpha+\beta x_1,\alpha+\beta x_2,...,\alpha+\beta x_n)^\top$ and $\Sigma = \sigma^2 I_n$.

The usual least squares estimator of $(\alpha,\beta)^\top$ can be written down and hence under the model assumptions, the joint distribution of $(\hat{\alpha},\hat{\beta})^\top$ and the marginal distributions of either obtained. Indeed because $(\hat{\alpha},\hat{\beta})^\top$ is linear in the Y's, they're bivariate normal and so the marginal distributions are also normal. It's a few lines of algebra to show that $E(\hat{\beta})=\beta$ and $\text{Var}(\hat{\beta})$ is as you give it.

When we carry out our exercise of drawing samples with the same x's, your estimate $b^{[1]}$, and my $b^{[2]}$ are just two draws from that normal distribution.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.