4
$\begingroup$

Under the condition of simple linear regression model ( $Y_i = \beta_0 + \beta_1X_i + \epsilon_i$) ordinary linear estimators($\hat{\beta_0}$ and $\hat{\beta_1}$) have minimum variance among all linear estimators.

To prove OLS estimator $\hat{\beta_1} = \sum{k_iy_i}$ has minimum variance we start by setting $\tilde{\beta_1} = \sum{c_iy_i}$ and we show that variance of $\tilde{\beta_1}$ can only be larger than $\beta_1$ if $c_i \neq k_i$.

Similarly, I am trying to prove that $\hat{\beta_0}$ has minimum variance among all unbiased linear estimators, and I am told that the proof starts similarly.

I know that the OLS estimator is $\hat{\beta_0} = \bar{y} - \hat{\beta_1}\bar{x}$.

How do I start the proof: by constructing another linear estimator $\tilde{\beta_0}$? Is this a linear estimator $\hat{\beta_0} = c\bar{y} - \hat{\beta_1}\bar{x}$?

$\endgroup$
6
  • $\begingroup$ "To prove OLS estimator $\hat{\beta_1} = \sum{k_iy_i}$ has minimum variance". You mean the minimum mean squared error. When you set $k_i = 0$ then the estimator has zero variance. en.wikipedia.org/wiki/Mean_squared_error en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff $\endgroup$ Feb 4 '20 at 15:46
  • $\begingroup$ @SextusEmpiricus - but $k_i = 0$ doesn't deliver an unbiased estimator. $\endgroup$
    – jbowman
    Feb 9 '20 at 17:22
  • $\begingroup$ @jbowman that's correct. When bias is zero then: error = bias + variance = variance $\endgroup$ Feb 9 '20 at 18:33
  • 1
    $\begingroup$ @SextusEmpiricus - but the OP is asking for a proof for unbiased estimators, so given that he's restricted it to that class... $\endgroup$
    – jbowman
    Feb 9 '20 at 18:35
  • $\begingroup$ yes correct, I had overlooked that. $\endgroup$ Feb 9 '20 at 21:40
2
+50
$\begingroup$

You can start by expressing $\hat{\beta_0}$ as a linear combination of $y_i$, similar to $\hat{\beta_1} = \sum{k_iy_i}$:

$\hat{\beta_0} = \bar{y} - \hat{\beta_1}\bar{x} = \frac{1}{N}\sum{y_i}-\sum{k_iy_i}\bar{x}=\sum{(\frac{1}{N}-k_i\bar{x})y_i}=\sum{l_iy_i}$

$\endgroup$
2
  • $\begingroup$ Thanks. I was able to figure it out myself. But this is what I was looking for. Also, can you please shade some light on why the estimator I proposed is not linear estimator? $\endgroup$ Feb 11 '20 at 3:07
  • 1
    $\begingroup$ It is, but it helps to write it out in terms of $y_i$ to see that it is very similar to $\hat{\beta_1}$. In the proof of the Gauss-Markov theorem using matrix notation there is no distinction between $\beta_0$ and $\beta_1$, they are lumped together in one $\beta$ matrix. $\endgroup$
    – Evgenia
    Feb 15 '20 at 19:05
2
$\begingroup$

The result you are trying to prove is called the Gauss-Markov theorem, and there are a number of available proofs you can find with a quick internet search. It can be proved using some simple matrix algebra. Since your goal is to prove the theorem yourself, I will not give you the full proof, but hopefully I can get you started, and give some general tips how to approach the problem. The usual starting point for the proof would be to assume a linear estimator of the form:

$$\hat{\boldsymbol{\beta}} = \mathbf{A} \mathbf{Y} \quad \quad \quad \quad \quad \mathbf{A} = (\mathbf{x}^\text{T} \mathbf{x})^{-1} \mathbf{x}^\text{T} + \mathbf{B}.$$

The ordinary least squares (OLS) estimator occurs in the case where $\mathbf{B}=\mathbf{0}$, and other linear estimators occur in the case where $\mathbf{B} \neq \mathbf{0}$. Now, from here you should be able to use some matrix algebra to obtain expressions for the mean and variance of the estimator, using the assumption that $\mathbf{Y}$ follows the form of the linear regression model. Unbiasedness requires that the mean be equal to the true parameter vector, and this should give you some requirement on the matrix $\mathbf{B}$. Using this condition, you should be able to simplify your variance expression. You then need to show that this variance expression is minimised when $\mathbf{B} = 0$, so that the OLS estimator is the MVLUE.

$\endgroup$
1
  • $\begingroup$ Thank you. But we have not gotten into Multiple Linear Regression yet. I need to know how can I solve it without using matrix notation. $\endgroup$ Feb 4 '20 at 18:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.