# How to prove $\beta_0$ has minimum variance among all unbiased linear estimator: Simple Linear Regression

Under the condition of simple linear regression model ( $$Y_i = \beta_0 + \beta_1X_i + \epsilon_i$$) ordinary linear estimators($$\hat{\beta_0}$$ and $$\hat{\beta_1}$$) have minimum variance among all linear estimators.

To prove OLS estimator $$\hat{\beta_1} = \sum{k_iy_i}$$ has minimum variance we start by setting $$\tilde{\beta_1} = \sum{c_iy_i}$$ and we show that variance of $$\tilde{\beta_1}$$ can only be larger than $$\beta_1$$ if $$c_i \neq k_i$$.

Similarly, I am trying to prove that $$\hat{\beta_0}$$ has minimum variance among all unbiased linear estimators, and I am told that the proof starts similarly.

I know that the OLS estimator is $$\hat{\beta_0} = \bar{y} - \hat{\beta_1}\bar{x}$$.

How do I start the proof: by constructing another linear estimator $$\tilde{\beta_0}$$? Is this a linear estimator $$\hat{\beta_0} = c\bar{y} - \hat{\beta_1}\bar{x}$$?

• "To prove OLS estimator $\hat{\beta_1} = \sum{k_iy_i}$ has minimum variance". You mean the minimum mean squared error. When you set $k_i = 0$ then the estimator has zero variance. en.wikipedia.org/wiki/Mean_squared_error en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff Feb 4 '20 at 15:46
• @SextusEmpiricus - but $k_i = 0$ doesn't deliver an unbiased estimator. Feb 9 '20 at 17:22
• @jbowman that's correct. When bias is zero then: error = bias + variance = variance Feb 9 '20 at 18:33
• @SextusEmpiricus - but the OP is asking for a proof for unbiased estimators, so given that he's restricted it to that class... Feb 9 '20 at 18:35
• yes correct, I had overlooked that. Feb 9 '20 at 21:40

You can start by expressing $$\hat{\beta_0}$$ as a linear combination of $$y_i$$, similar to $$\hat{\beta_1} = \sum{k_iy_i}$$:

$$\hat{\beta_0} = \bar{y} - \hat{\beta_1}\bar{x} = \frac{1}{N}\sum{y_i}-\sum{k_iy_i}\bar{x}=\sum{(\frac{1}{N}-k_i\bar{x})y_i}=\sum{l_iy_i}$$

• Thanks. I was able to figure it out myself. But this is what I was looking for. Also, can you please shade some light on why the estimator I proposed is not linear estimator? Feb 11 '20 at 3:07
• It is, but it helps to write it out in terms of $y_i$ to see that it is very similar to $\hat{\beta_1}$. In the proof of the Gauss-Markov theorem using matrix notation there is no distinction between $\beta_0$ and $\beta_1$, they are lumped together in one $\beta$ matrix. Feb 15 '20 at 19:05

The result you are trying to prove is called the Gauss-Markov theorem, and there are a number of available proofs you can find with a quick internet search. It can be proved using some simple matrix algebra. Since your goal is to prove the theorem yourself, I will not give you the full proof, but hopefully I can get you started, and give some general tips how to approach the problem. The usual starting point for the proof would be to assume a linear estimator of the form:

$$\hat{\boldsymbol{\beta}} = \mathbf{A} \mathbf{Y} \quad \quad \quad \quad \quad \mathbf{A} = (\mathbf{x}^\text{T} \mathbf{x})^{-1} \mathbf{x}^\text{T} + \mathbf{B}.$$

The ordinary least squares (OLS) estimator occurs in the case where $$\mathbf{B}=\mathbf{0}$$, and other linear estimators occur in the case where $$\mathbf{B} \neq \mathbf{0}$$. Now, from here you should be able to use some matrix algebra to obtain expressions for the mean and variance of the estimator, using the assumption that $$\mathbf{Y}$$ follows the form of the linear regression model. Unbiasedness requires that the mean be equal to the true parameter vector, and this should give you some requirement on the matrix $$\mathbf{B}$$. Using this condition, you should be able to simplify your variance expression. You then need to show that this variance expression is minimised when $$\mathbf{B} = 0$$, so that the OLS estimator is the MVLUE.

• Thank you. But we have not gotten into Multiple Linear Regression yet. I need to know how can I solve it without using matrix notation. Feb 4 '20 at 18:43