1
$\begingroup$

I would like to show that the ridge regression estimator:

$$\beta^R = (X^TX+\lambda I)^{-1}X^T Y$$

is biased, where $Y \sim N(X\beta, \sigma^2 I)$.

If we assume that $X^TX$ is invertible, this can be done by writing it in the form

$$\beta^{R} = (I+\lambda (X^TX)^{-1}) \beta^{LS}$$

and then using the fact that the least squares estimator is unbiased.

Since we cannot always assume $X^TX$ is invertible, say for example when $X$ is sparse, it seems we must instead deal with generalised inverse $G$ of $X^TX$ where

$$(X^TX)G(X^TX)=X^TX$$

To use the same approach, noting that in the case when $X$ is not of full rank, $\beta^{LS}$ is given by

$$\beta^{LS} = GX^TY$$

And $X\beta^{LS}$ is an unbiased estimator of $X\beta$.

However, I have been unable to do so.

$\endgroup$
  • 1
    $\begingroup$ When $X^\text{T}X$ is not invertible the least square estimator is not unique and not unbiased. $\endgroup$ – Xi'an Apr 18 at 5:18
  • 1
    $\begingroup$ I think you can check that unbiasedness would imply that $X^\top X$ is invertible, with inverse $(X^\top X + \lambda I)^{-1}$. But we know this is not true. Note that unbiasedness is a property of the estimator for all $\beta$. $\endgroup$ – guy Apr 18 at 6:20
  • $\begingroup$ When $X^TX$ is not invertible, "the" ols estimator is not unique, so to make the question meaningful you could specify one of the infinitely many possible solutions. $\endgroup$ – kjetil b halvorsen Apr 18 at 19:22
  • $\begingroup$ @guy thanks for the hint. I believe that by assuming it is unbiased, I have shown this implies $\lambda = 0$, so that it must be the least squares estimator. Thank you! $\endgroup$ – Xiaomi Apr 20 at 2:57
1
$\begingroup$

I think the answer is actually quite elementary, but someone may correct me if I have misunderstood something.

Suppose that the estimator is unbiased, so that we have

$$E[\beta^R] = \beta$$

for all $\beta$. Then, since $E[Y] = X\beta$, this implies that

$$(X^TX+\lambda I)^{-1}X^TX \beta = \beta$$

Multiplying both sides by $$(X^TX+\lambda I)$$ then yields

$$X^TX \beta = X^TX\beta + \lambda \beta$$

Hence

$$\lambda \beta = 0$$

Since this must hold for all $\beta$, this implies $\lambda = 0$. Hence, only the least squares estimate when $\lambda = 0$ is unbiased.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.