I am confused about whether the value of the sampling variance of the OLS estimator of a regression coefficient (e.g. slope) differs from sample to sample.

Assume we have the following simple linear regression model:

$$Y = \beta_0 + \beta_1X + \epsilon$$ where $\epsilon\sim N(0, \sigma^2)$.

Let $\hat{\beta}_1$ be the OLS estimator of the slope $\beta_1$.

I know that the variance of the sampling distribution of the the OLS estimator $\hat{\beta}_1$ is given by the following:

$Var[\hat{\beta}_1]=\frac{\sigma^2}{\sum_{i = 1}^{n}(x_i - \bar{x})}$

I am confused because apparently $Var[\hat{\beta}_1]$ is a function of the sample mean $\bar{x}$ which differs from sample to sample, does that mean $Var[\hat{\beta}_1]$ also differs from sample to sample?

In addition, I am wondering how to calculate the sample estimate of $Var[\hat{\beta}_1]$. My understanding is a sample estimate of the true variance of the sampling distribution of $\hat{\beta}_1$ is given by replacing the true error variance $\sigma^2 = Var[\epsilon]$ with the variance of the residual $\hat{\sigma}^2 = Var[\hat{\epsilon}]$ where $\hat{\epsilon} = Y - \hat{Y}$. Is that correct? If so, then the sample estimate of $Var[\hat{\beta}_1]$ is written as:

$$\widehat{Var}[\hat{\beta}_1] = \frac{\hat{\sigma}^2}{\sum_{i = 1}^{n}(x_i - \bar{x})}$$

But the value of the true sampling variance $Var[\hat{\beta}_1]$ differs from sample to sample, then how can we calculate the bias of the sample estimate $\widehat{Var}[\hat{\beta}_1]$ since the bias would be:

$Bias[\widehat{Var}[\hat{\beta}_1]] = E[\widehat{Var}[\hat{\beta}_1]] - Var[\hat{\beta}_1]$

Any suggestion/comment is welcomed!

So when I am trying to assess the bias of the sample estimator, $\widehat{Var}[\hat{\beta}_1]$ by simulation.

My understanding of the simulation process is:

  1. generate a sample of x and y, regress y on x and get the residual variance $\hat{\sigma}^2$ and use it and the sampled Xs to calculate $\widehat{Var}[\hat{\beta}_1]$.

  2. repeatedly generate K such samples, and for each sample I can get a different residual variance $\hat{\sigma}_1^2$,...,$\hat{\sigma}_K^2$ and K sets of Xs, $X_1,...,X_k$, and consequently K estimates $\widehat{Var}_1[\hat{\beta}_1],...,\widehat{Var}_K[\hat{\beta}_1]$.

  3. I then take the expectation of these K estimates $E[\widehat{Var}[\hat{\beta}_1]]=\frac{1}{K}\sum_{i=1}^{K}\widehat{Var}_i[\hat{\beta}_1]$.

  4. Now to determine how much empirical bias I have, I have to compare this expectation to true sampling variance of $\hat{\beta}_1$, whcih is $Var[\hat{\beta}_1]$. But how do I calculate $Var[\hat{\beta}_1]$ if for each of the sample K, $Var[\hat{\beta}_1]$ value differs since although $\sigma^2$ (the true error variance) does not differ, the sample mean of $X$, $\bar{x}$ differs. Based on what you said, it seems that the true variance $Var[\hat{\beta}_1]$ should be calculated using the set of Xs and $\bar{x}$ that comes from the first of the K samples (i.e. sample 1)? Am I completely in the wrong direction?

  • $\begingroup$ I think you have an error because $\sum^n_{i=1} (x_i - \bar{x}) = 0$ always. $\endgroup$
    – Noah
    Commented Mar 1 at 18:46
  • $\begingroup$ Assuming "sample to sample" means you hold all the $x_i$ fixed at the same values then the variance does not change -- but (of course) any estimator of that variance is likely to change because it's a function of the random responses Please, then, explain more fully what you mean by "from sample to sample". $\endgroup$
    – whuber
    Commented Mar 1 at 19:18
  • $\begingroup$ I am trying to assess the bias of $\widehat{Var}[\hat{\beta}_1]$ by simulating K samples and for each sample I calculate $\widehat{Var}[\hat{\beta}_1]$ and then take the expectation of these K estimators. Then to find the bias, I need to compare it to the true variance, $Var[\hat{\beta}_1]$, but if $Var[\hat{\beta}_1]$ has a different value in each of the K sample, then I do not know what I am comparing the expectation of the K estimator, $\widehat{Var}[\hat{\beta}_1]$, to. I hope I have made this a bit clear. $\endgroup$ Commented Mar 1 at 23:46

1 Answer 1


In the OLS model, the $x_i$ are viewed as fixed/non-random, and the $y_i$ are viewed as random (due to the noise from $\epsilon_i$).

$\hat{\beta_1}$ is a function of the data $(x_i, y_i)$, but the variance $\text{Var}(\hat{\beta}_1)$ is considering only the variance coming from the $y_i$ (via $\epsilon_i$). So it is natural that the expression for the variance is still a function of the fixed/non-random $x_i$. To flesh out whuber's comment,

  • If you keep the $x_i$ the same, but generate new draws of the $y_i$ from the OLS model to obtain a new dataset $(x_i, y'_i)$, then the variance of $\hat{\beta}_1$ is still $\sigma^2 / \sum_{i=1}^n (x_i - \bar{x})^2$, the same as for your original dataset $(x_i, y_i)$.
  • If however you have an entirely new dataset $(x'_i, y'_i)$, then the variance would be different: $\sigma^2 / \sum_{i=1}^n (x'_i - \bar{x}')^2$.

The vector of residuals $\hat{\epsilon}$ can be written as $\hat{\epsilon} = Y - \hat{Y} = (I - H) Y = (I-H)\epsilon$ where $X = \begin{bmatrix} 1 & x_1 \\ \vdots & \vdots \\ 1 & x_n\end{bmatrix}$ and $H = X(X^\top X)^{-1} X^\top$. From this one can show that $\hat{\epsilon}$ is multivariate normal with mean zero and covariance matrix $\sigma^2 (I-H)^2 = \sigma^2(I-H)$.

However, $\hat{\epsilon}^\top \hat{\epsilon} = \epsilon^\top (I-H)^2 \epsilon = \epsilon^\top (I-H) \epsilon = \text{Tr}((I-H)\epsilon \epsilon^\top)$ has expectation $E[\hat{\epsilon}^\top \hat{\epsilon}] = \text{Tr}((I-H)E[\epsilon \epsilon^\top]) = \text{Tr}(\sigma^2 (I-H)) = (n-2)\sigma^2$, so $$\hat{\sigma}^2 := \frac{1}{n-2} \sum_{i=1}^n \hat{\epsilon}_i^2$$ is an unbiased estimator of $\sigma^2$. This is essentially the sample variance of the residuals (which I guess is what you meant by "$\text{Var}(\hat{\epsilon})$"), but note the normalization factor $\frac{1}{n-2}$. In general if you have $p$ covariates and an intercept in your linear model, the normalization factor is $\frac{1}{n-p-1}$.

This can then be used to get an unbiased estimator of $\text{Var}(\hat{\beta}_1)$ via $\hat{\sigma}^2 / \sum_{i=1}^n (x_i - \bar{x})^2$ as you mentioned.

  • $\begingroup$ Thanks for replying. It is helpful but I am still a bit confused. I wrote my question as an answer since stackexchange does not allow long comments. $\endgroup$ Commented Mar 2 at 0:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.