Sampling Variance of OLS Estimators of Regression Coefficients

Question

I am confused about whether the value of the sampling variance of the OLS estimator of a regression coefficient (e.g. slope) differs from sample to sample.

Assume we have the following simple linear regression model:

$$Y = \beta_0 + \beta_1X + \epsilon$$ where $\epsilon\sim N(0, \sigma^2)$.

Let $\hat{\beta}_1$ be the OLS estimator of the slope $\beta_1$.

I know that the variance of the sampling distribution of the the OLS estimator $\hat{\beta}_1$ is given by the following:

$Var[\hat{\beta}_1]=\frac{\sigma^2}{\sum_{i = 1}^{n}(x_i - \bar{x})}$

I am confused because apparently $Var[\hat{\beta}_1]$ is a function of the sample mean $\bar{x}$ which differs from sample to sample, does that mean $Var[\hat{\beta}_1]$ also differs from sample to sample?

In addition, I am wondering how to calculate the sample estimate of $Var[\hat{\beta}_1]$. My understanding is a sample estimate of the true variance of the sampling distribution of $\hat{\beta}_1$ is given by replacing the true error variance $\sigma^2 = Var[\epsilon]$ with the variance of the residual $\hat{\sigma}^2 = Var[\hat{\epsilon}]$ where $\hat{\epsilon} = Y - \hat{Y}$. Is that correct? If so, then the sample estimate of $Var[\hat{\beta}_1]$ is written as:

$$\widehat{Var}[\hat{\beta}_1] = \frac{\hat{\sigma}^2}{\sum_{i = 1}^{n}(x_i - \bar{x})}$$

But the value of the true sampling variance $Var[\hat{\beta}_1]$ differs from sample to sample, then how can we calculate the bias of the sample estimate $\widehat{Var}[\hat{\beta}_1]$ since the bias would be:

$Bias[\widehat{Var}[\hat{\beta}_1]] = E[\widehat{Var}[\hat{\beta}_1]] - Var[\hat{\beta}_1]$

Any suggestion/comment is welcomed!

---

So when I am trying to assess the bias of the sample estimator, $\widehat{Var}[\hat{\beta}_1]$ by simulation.

My understanding of the simulation process is:

1. generate a sample of x and y, regress y on x and get the residual variance $\hat{\sigma}^2$ and use it and the sampled Xs to calculate $\widehat{Var}[\hat{\beta}_1]$.

2. repeatedly generate K such samples, and for each sample I can get a different residual variance $\hat{\sigma}_1^2$,...,$\hat{\sigma}_K^2$ and K sets of Xs, $X_1,...,X_k$, and consequently K estimates $\widehat{Var}_1[\hat{\beta}_1],...,\widehat{Var}_K[\hat{\beta}_1]$.

3. I then take the expectation of these K estimates $E[\widehat{Var}[\hat{\beta}_1]]=\frac{1}{K}\sum_{i=1}^{K}\widehat{Var}_i[\hat{\beta}_1]$.

4. Now to determine how much empirical bias I have, I have to compare this expectation to true sampling variance of $\hat{\beta}_1$, whcih is $Var[\hat{\beta}_1]$. But how do I calculate $Var[\hat{\beta}_1]$ if for each of the sample K, $Var[\hat{\beta}_1]$ value differs since although $\sigma^2$ (the true error variance) does not differ, the sample mean of $X$, $\bar{x}$ differs. Based on what you said, it seems that the true variance $Var[\hat{\beta}_1]$ should be calculated using the set of Xs and $\bar{x}$ that comes from the first of the K samples (i.e. sample 1)? Am I completely in the wrong direction?

Answer 1

In the OLS model, the $x_i$ are viewed as fixed/non-random, and the $y_i$ are viewed as random (due to the noise from $\epsilon_i$).

$\hat{\beta_1}$ is a function of the data $(x_i, y_i)$, but the variance $\text{Var}(\hat{\beta}_1)$ is considering only the variance coming from the $y_i$ (via $\epsilon_i$). So it is natural that the expression for the variance is still a function of the fixed/non-random $x_i$. To flesh out whuber's comment,

• If you keep the $x_i$ the same, but generate new draws of the $y_i$ from the OLS model to obtain a new dataset $(x_i, y'_i)$, then the variance of $\hat{\beta}_1$ is still $\sigma^2 / \sum_{i=1}^n (x_i - \bar{x})^2$, the same as for your original dataset $(x_i, y_i)$.
• If however you have an entirely new dataset $(x'_i, y'_i)$, then the variance would be different: $\sigma^2 / \sum_{i=1}^n (x'_i - \bar{x}')^2$.

---

The vector of residuals $\hat{\epsilon}$ can be written as $\hat{\epsilon} = Y - \hat{Y} = (I - H) Y = (I-H)\epsilon$ where $X = \begin{bmatrix} 1 & x_1 \\ \vdots & \vdots \\ 1 & x_n\end{bmatrix}$ and $H = X(X^\top X)^{-1} X^\top$. From this one can show that $\hat{\epsilon}$ is multivariate normal with mean zero and covariance matrix $\sigma^2 (I-H)^2 = \sigma^2(I-H)$.

However, $\hat{\epsilon}^\top \hat{\epsilon} = \epsilon^\top (I-H)^2 \epsilon = \epsilon^\top (I-H) \epsilon = \text{Tr}((I-H)\epsilon \epsilon^\top)$ has expectation $E[\hat{\epsilon}^\top \hat{\epsilon}] = \text{Tr}((I-H)E[\epsilon \epsilon^\top]) = \text{Tr}(\sigma^2 (I-H)) = (n-2)\sigma^2$, so $$\hat{\sigma}^2 := \frac{1}{n-2} \sum_{i=1}^n \hat{\epsilon}_i^2$$ is an unbiased estimator of $\sigma^2$. This is essentially the sample variance of the residuals (which I guess is what you meant by "$\text{Var}(\hat{\epsilon})$"), but note the normalization factor $\frac{1}{n-2}$. In general if you have $p$ covariates and an intercept in your linear model, the normalization factor is $\frac{1}{n-p-1}$.

This can then be used to get an unbiased estimator of $\text{Var}(\hat{\beta}_1)$ via $\hat{\sigma}^2 / \sum_{i=1}^n (x_i - \bar{x})^2$ as you mentioned.