5
$\begingroup$

For least squares estimation, the estimator $\hat{\beta} = X^{T}Y$ is an unbiased estimator while $\hat{\beta} = (X^{T}X)^{-1}X^{T}Y$ is also an unbiased estimator given that $X$ is well-defined. Is there a difference between these two estimators through adding on the $(X^{T}X)^{-1}$ term? Thanks.

$\endgroup$
1
  • 4
    $\begingroup$ Your first estimator is only unbiased if $X$ matrix is orthonormal, i.e. $X^\intercal X = I$, the identity. Otherwise it is biased. $\endgroup$
    – bdeonovic
    Feb 2, 2016 at 14:23

2 Answers 2

11
$\begingroup$

I don't believe that the first estimator is unbiased.

Under the linear model assumption

$$ Y = X \beta + \epsilon $$

the expectation of the first estimator is

$$ E[X^{T} Y] = E[X^{T}X \beta] + E[X^{T} \epsilon] = \underbrace{X^{T}X \beta + X^{T} E[\epsilon]}_{\text{linearity of expectation}} = X^{T}X \beta$$

From which we conclude that the proposed estimator is unbiased if and only if $X^{T} X \beta = \beta$.

This computation also clearly shows why the second estimator is always unbiased.

$\endgroup$
9
$\begingroup$

Your first estimator doesn't work. It's not only biased, but the bias increases with the sample size.

Imagine a simple intercept model: $$y_i=c+\varepsilon_i$$ here your design matrix is a simple vector of ones, $x_i=1$.

Your estimator is $$\hat c = X'Y\equiv\sum_{i=1}^n y_i$$

If you add more data your estimator keeps increasing without bound: $$\hat c\to\infty$$ when $n\to\infty$

This doesn't make a sense, because, $E[y_i]=c$, assuming $E[\varepsilon_i]=0$, which suggests that maybe $\hat c=\bar y_i$ would be a good estimator.

However, here $X'X=n$, and with the second estimator you get what you'd expect from a reasonable looking estimator: $$\hat c=\frac{1}{n}\sum_{i=1}^n y_i\equiv \bar y$$

UPDATE: Your second estimator will work in one case: if the design matrix is orthonormal and square, i.e. forms orthonormal basis.

The simplest case is when you have exactly one observation in my example. You have $X=1$, so $X'Y=y_1$, then you get: $$\hat c=y_1$$

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.