# Showing estimator is biased without assuming $X^TX$ is invertible?

I would like to show that the ridge regression estimator:

$$\beta^R = (X^TX+\lambda I)^{-1}X^T Y$$

is biased, where $$Y \sim N(X\beta, \sigma^2 I)$$.

If we assume that $$X^TX$$ is invertible, this can be done by writing it in the form

$$\beta^{R} = (I+\lambda (X^TX)^{-1}) \beta^{LS}$$

and then using the fact that the least squares estimator is unbiased.

Since we cannot always assume $$X^TX$$ is invertible, say for example when $$X$$ is sparse, it seems we must instead deal with generalised inverse $$G$$ of $$X^TX$$ where

$$(X^TX)G(X^TX)=X^TX$$

To use the same approach, noting that in the case when $$X$$ is not of full rank, $$\beta^{LS}$$ is given by

$$\beta^{LS} = GX^TY$$

And $$X\beta^{LS}$$ is an unbiased estimator of $$X\beta$$.

However, I have been unable to do so.

• When $X^\text{T}X$ is not invertible the least square estimator is not unique and not unbiased. – Xi'an Apr 18 at 5:18
• I think you can check that unbiasedness would imply that $X^\top X$ is invertible, with inverse $(X^\top X + \lambda I)^{-1}$. But we know this is not true. Note that unbiasedness is a property of the estimator for all $\beta$. – guy Apr 18 at 6:20
• When $X^TX$ is not invertible, "the" ols estimator is not unique, so to make the question meaningful you could specify one of the infinitely many possible solutions. – kjetil b halvorsen Apr 18 at 19:22
• @guy thanks for the hint. I believe that by assuming it is unbiased, I have shown this implies $\lambda = 0$, so that it must be the least squares estimator. Thank you! – Xiaomi Apr 20 at 2:57

I think the answer is actually quite elementary, but someone may correct me if I have misunderstood something.

Suppose that the estimator is unbiased, so that we have

$$E[\beta^R] = \beta$$

for all $$\beta$$. Then, since $$E[Y] = X\beta$$, this implies that

$$(X^TX+\lambda I)^{-1}X^TX \beta = \beta$$

Multiplying both sides by $$(X^TX+\lambda I)$$ then yields

$$X^TX \beta = X^TX\beta + \lambda \beta$$

Hence

$$\lambda \beta = 0$$

Since this must hold for all $$\beta$$, this implies $$\lambda = 0$$. Hence, only the least squares estimate when $$\lambda = 0$$ is unbiased.