Under the condition of simple linear regression model ( $Y_i = \beta_0 + \beta_1X_i + \epsilon_i$) ordinary linear estimators($\hat{\beta_0}$ and $\hat{\beta_1}$) have minimum variance among all linear estimators.

To prove OLS estimator $\hat{\beta_1} = \sum{k_iy_i}$ has minimum variance we start by setting $\tilde{\beta_1} = \sum{c_iy_i}$ and we show that variance of $\tilde{\beta_1}$ can only be larger than $\beta_1$ if $c_i \neq k_i$.

Similarly, I am trying to prove that $\hat{\beta_0}$ has minimum variance among all unbiased linear estimators, and I am told that the proof starts similarly.

I know that the OLS estimator is $\hat{\beta_0} = \bar{y} - \hat{\beta_1}\bar{x}$.

How do I start the proof: by constructing another linear estimator $\tilde{\beta_0}$? Is this a linear estimator $\hat{\beta_0} = c\bar{y} - \hat{\beta_1}\bar{x}$?

  • $\begingroup$ "To prove OLS estimator $\hat{\beta_1} = \sum{k_iy_i}$ has minimum variance". You mean the minimum mean squared error. When you set $k_i = 0$ then the estimator has zero variance. en.wikipedia.org/wiki/Mean_squared_error en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff $\endgroup$ – Sextus Empiricus Feb 4 at 15:46
  • $\begingroup$ @SextusEmpiricus - but $k_i = 0$ doesn't deliver an unbiased estimator. $\endgroup$ – jbowman Feb 9 at 17:22
  • $\begingroup$ @jbowman that's correct. When bias is zero then: error = bias + variance = variance $\endgroup$ – Sextus Empiricus Feb 9 at 18:33
  • 1
    $\begingroup$ @SextusEmpiricus - but the OP is asking for a proof for unbiased estimators, so given that he's restricted it to that class... $\endgroup$ – jbowman Feb 9 at 18:35
  • $\begingroup$ yes correct, I had overlooked that. $\endgroup$ – Sextus Empiricus Feb 9 at 21:40

You can start by expressing $\hat{\beta_0}$ as a linear combination of $y_i$, similar to $\hat{\beta_1} = \sum{k_iy_i}$:

$\hat{\beta_0} = \bar{y} - \hat{\beta_1}\bar{x} = \frac{1}{N}\sum{y_i}-\sum{k_iy_i}\bar{x}=\sum{(\frac{1}{N}-k_i\bar{x})y_i}=\sum{l_iy_i}$

| cite | improve this answer | |
  • $\begingroup$ Thanks. I was able to figure it out myself. But this is what I was looking for. Also, can you please shade some light on why the estimator I proposed is not linear estimator? $\endgroup$ – Quazi Irfan Feb 11 at 3:07
  • 1
    $\begingroup$ It is, but it helps to write it out in terms of $y_i$ to see that it is very similar to $\hat{\beta_1}$. In the proof of the Gauss-Markov theorem using matrix notation there is no distinction between $\beta_0$ and $\beta_1$, they are lumped together in one $\beta$ matrix. $\endgroup$ – Evgenia Feb 15 at 19:05

The result you are trying to prove is called the Gauss-Markov theorem, and there are a number of available proofs you can find with a quick internet search. It can be proved using some simple matrix algebra. Since your goal is to prove the theorem yourself, I will not give you the full proof, but hopefully I can get you started, and give some general tips how to approach the problem. The usual starting point for the proof would be to assume a linear estimator of the form:

$$\hat{\boldsymbol{\beta}} = \mathbf{A} \mathbf{Y} \quad \quad \quad \quad \quad \mathbf{A} = (\mathbf{x}^\text{T} \mathbf{x})^{-1} \mathbf{x}^\text{T} + \mathbf{B}.$$

The ordinary least squares (OLS) estimator occurs in the case where $\mathbf{B}=\mathbf{0}$, and other linear estimators occur in the case where $\mathbf{B} \neq \mathbf{0}$. Now, from here you should be able to use some matrix algebra to obtain expressions for the mean and variance of the estimator, using the assumption that $\mathbf{Y}$ follows the form of the linear regression model. Unbiasedness requires that the mean be equal to the true parameter vector, and this should give you some requirement on the matrix $\mathbf{B}$. Using this condition, you should be able to simplify your variance expression. You then need to show that this variance expression is minimised when $\mathbf{B} = 0$, so that the OLS estimator is the MVLUE.

| cite | improve this answer | |
  • $\begingroup$ Thank you. But we have not gotten into Multiple Linear Regression yet. I need to know how can I solve it without using matrix notation. $\endgroup$ – Quazi Irfan Feb 4 at 18:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.