For least squares estimation, what is the difference between using the estimator $\hat{\beta} = X^{T}Y$ vs $\hat{\beta} = (X^{T}X)^{-1}X^{T}Y$

Question

For least squares estimation, the estimator $\hat{\beta} = X^{T}Y$ is an unbiased estimator while $\hat{\beta} = (X^{T}X)^{-1}X^{T}Y$ is also an unbiased estimator given that $X$ is well-defined. Is there a difference between these two estimators through adding on the $(X^{T}X)^{-1}$ term? Thanks.

Answer 1

I don't believe that the first estimator is unbiased.

Under the linear model assumption

$$ Y = X \beta + \epsilon $$

the expectation of the first estimator is

$$ E[X^{T} Y] = E[X^{T}X \beta] + E[X^{T} \epsilon] = \underbrace{X^{T}X \beta + X^{T} E[\epsilon]}_{\text{linearity of expectation}} = X^{T}X \beta$$

From which we conclude that the proposed estimator is unbiased if and only if $X^{T} X \beta = \beta$.

This computation also clearly shows why the second estimator is always unbiased.

Answer 2

Your first estimator doesn't work. It's not only biased, but the bias increases with the sample size.

Imagine a simple intercept model: $$y_i=c+\varepsilon_i$$ here your design matrix is a simple vector of ones, $x_i=1$.

Your estimator is $$\hat c = X'Y\equiv\sum_{i=1}^n y_i$$

If you add more data your estimator keeps increasing without bound: $$\hat c\to\infty$$ when $n\to\infty$

This doesn't make a sense, because, $E[y_i]=c$, assuming $E[\varepsilon_i]=0$, which suggests that maybe $\hat c=\bar y_i$ would be a good estimator.

However, here $X'X=n$, and with the second estimator you get what you'd expect from a reasonable looking estimator: $$\hat c=\frac{1}{n}\sum_{i=1}^n y_i\equiv \bar y$$

UPDATE: Your second estimator will work in one case: if the design matrix is orthonormal and square, i.e. forms orthonormal basis.

The simplest case is when you have exactly one observation in my example. You have $X=1$, so $X'Y=y_1$, then you get: $$\hat c=y_1$$