# Sampling Variance of OLS Estimators of Regression Coefficients

I am confused about whether the value of the sampling variance of the OLS estimator of a regression coefficient (e.g. slope) differs from sample to sample.

Assume we have the following simple linear regression model:

$$Y = \beta_0 + \beta_1X + \epsilon$$ where $$\epsilon\sim N(0, \sigma^2)$$.

Let $$\hat{\beta}_1$$ be the OLS estimator of the slope $$\beta_1$$.

I know that the variance of the sampling distribution of the the OLS estimator $$\hat{\beta}_1$$ is given by the following:

$$Var[\hat{\beta}_1]=\frac{\sigma^2}{\sum_{i = 1}^{n}(x_i - \bar{x})}$$

I am confused because apparently $$Var[\hat{\beta}_1]$$ is a function of the sample mean $$\bar{x}$$ which differs from sample to sample, does that mean $$Var[\hat{\beta}_1]$$ also differs from sample to sample?

In addition, I am wondering how to calculate the sample estimate of $$Var[\hat{\beta}_1]$$. My understanding is a sample estimate of the true variance of the sampling distribution of $$\hat{\beta}_1$$ is given by replacing the true error variance $$\sigma^2 = Var[\epsilon]$$ with the variance of the residual $$\hat{\sigma}^2 = Var[\hat{\epsilon}]$$ where $$\hat{\epsilon} = Y - \hat{Y}$$. Is that correct? If so, then the sample estimate of $$Var[\hat{\beta}_1]$$ is written as:

$$\widehat{Var}[\hat{\beta}_1] = \frac{\hat{\sigma}^2}{\sum_{i = 1}^{n}(x_i - \bar{x})}$$

But the value of the true sampling variance $$Var[\hat{\beta}_1]$$ differs from sample to sample, then how can we calculate the bias of the sample estimate $$\widehat{Var}[\hat{\beta}_1]$$ since the bias would be:

$$Bias[\widehat{Var}[\hat{\beta}_1]] = E[\widehat{Var}[\hat{\beta}_1]] - Var[\hat{\beta}_1]$$

Any suggestion/comment is welcomed!

So when I am trying to assess the bias of the sample estimator, $$\widehat{Var}[\hat{\beta}_1]$$ by simulation.

My understanding of the simulation process is:

1. generate a sample of x and y, regress y on x and get the residual variance $$\hat{\sigma}^2$$ and use it and the sampled Xs to calculate $$\widehat{Var}[\hat{\beta}_1]$$.

2. repeatedly generate K such samples, and for each sample I can get a different residual variance $$\hat{\sigma}_1^2$$,...,$$\hat{\sigma}_K^2$$ and K sets of Xs, $$X_1,...,X_k$$, and consequently K estimates $$\widehat{Var}_1[\hat{\beta}_1],...,\widehat{Var}_K[\hat{\beta}_1]$$.

3. I then take the expectation of these K estimates $$E[\widehat{Var}[\hat{\beta}_1]]=\frac{1}{K}\sum_{i=1}^{K}\widehat{Var}_i[\hat{\beta}_1]$$.

4. Now to determine how much empirical bias I have, I have to compare this expectation to true sampling variance of $$\hat{\beta}_1$$, whcih is $$Var[\hat{\beta}_1]$$. But how do I calculate $$Var[\hat{\beta}_1]$$ if for each of the sample K, $$Var[\hat{\beta}_1]$$ value differs since although $$\sigma^2$$ (the true error variance) does not differ, the sample mean of $$X$$, $$\bar{x}$$ differs. Based on what you said, it seems that the true variance $$Var[\hat{\beta}_1]$$ should be calculated using the set of Xs and $$\bar{x}$$ that comes from the first of the K samples (i.e. sample 1)? Am I completely in the wrong direction?

• I think you have an error because $\sum^n_{i=1} (x_i - \bar{x}) = 0$ always.
– Noah
Commented Mar 1 at 18:46
• Assuming "sample to sample" means you hold all the $x_i$ fixed at the same values then the variance does not change -- but (of course) any estimator of that variance is likely to change because it's a function of the random responses Please, then, explain more fully what you mean by "from sample to sample".
– whuber
Commented Mar 1 at 19:18
• I am trying to assess the bias of $\widehat{Var}[\hat{\beta}_1]$ by simulating K samples and for each sample I calculate $\widehat{Var}[\hat{\beta}_1]$ and then take the expectation of these K estimators. Then to find the bias, I need to compare it to the true variance, $Var[\hat{\beta}_1]$, but if $Var[\hat{\beta}_1]$ has a different value in each of the K sample, then I do not know what I am comparing the expectation of the K estimator, $\widehat{Var}[\hat{\beta}_1]$, to. I hope I have made this a bit clear. Commented Mar 1 at 23:46

In the OLS model, the $$x_i$$ are viewed as fixed/non-random, and the $$y_i$$ are viewed as random (due to the noise from $$\epsilon_i$$).

$$\hat{\beta_1}$$ is a function of the data $$(x_i, y_i)$$, but the variance $$\text{Var}(\hat{\beta}_1)$$ is considering only the variance coming from the $$y_i$$ (via $$\epsilon_i$$). So it is natural that the expression for the variance is still a function of the fixed/non-random $$x_i$$. To flesh out whuber's comment,

• If you keep the $$x_i$$ the same, but generate new draws of the $$y_i$$ from the OLS model to obtain a new dataset $$(x_i, y'_i)$$, then the variance of $$\hat{\beta}_1$$ is still $$\sigma^2 / \sum_{i=1}^n (x_i - \bar{x})^2$$, the same as for your original dataset $$(x_i, y_i)$$.
• If however you have an entirely new dataset $$(x'_i, y'_i)$$, then the variance would be different: $$\sigma^2 / \sum_{i=1}^n (x'_i - \bar{x}')^2$$.

The vector of residuals $$\hat{\epsilon}$$ can be written as $$\hat{\epsilon} = Y - \hat{Y} = (I - H) Y = (I-H)\epsilon$$ where $$X = \begin{bmatrix} 1 & x_1 \\ \vdots & \vdots \\ 1 & x_n\end{bmatrix}$$ and $$H = X(X^\top X)^{-1} X^\top$$. From this one can show that $$\hat{\epsilon}$$ is multivariate normal with mean zero and covariance matrix $$\sigma^2 (I-H)^2 = \sigma^2(I-H)$$.

However, $$\hat{\epsilon}^\top \hat{\epsilon} = \epsilon^\top (I-H)^2 \epsilon = \epsilon^\top (I-H) \epsilon = \text{Tr}((I-H)\epsilon \epsilon^\top)$$ has expectation $$E[\hat{\epsilon}^\top \hat{\epsilon}] = \text{Tr}((I-H)E[\epsilon \epsilon^\top]) = \text{Tr}(\sigma^2 (I-H)) = (n-2)\sigma^2$$, so $$\hat{\sigma}^2 := \frac{1}{n-2} \sum_{i=1}^n \hat{\epsilon}_i^2$$ is an unbiased estimator of $$\sigma^2$$. This is essentially the sample variance of the residuals (which I guess is what you meant by "$$\text{Var}(\hat{\epsilon})$$"), but note the normalization factor $$\frac{1}{n-2}$$. In general if you have $$p$$ covariates and an intercept in your linear model, the normalization factor is $$\frac{1}{n-p-1}$$.

This can then be used to get an unbiased estimator of $$\text{Var}(\hat{\beta}_1)$$ via $$\hat{\sigma}^2 / \sum_{i=1}^n (x_i - \bar{x})^2$$ as you mentioned.

• Thanks for replying. It is helpful but I am still a bit confused. I wrote my question as an answer since stackexchange does not allow long comments. Commented Mar 2 at 0:02