4
$\begingroup$

Consider the following estimator $$\hat\beta = \left(\sum_{i=1}^{N}x_ix_i' + \lambda I_k\right)^{-1}\left(\sum_{i=1}^Nx_iy_i\right)$$ where $x_i$ is a column vector $k\times1$ from $X$ and $\lambda > 0$ is a scalar and $\mathbb{E}(x_ie_i) = 0$ .

  1. Define bias and show that $\hat\beta$ is biased
  2. Define consistency and show that $\hat \beta$ is consistent
  3. Define conditional variance of $\hat\beta$.

For number 1 I have \begin{equation} \begin{aligned} \hat\beta &= \left(\sum_{i=1}^{N}x_ix_i' + \lambda I_k\right)^{-1}\left(\sum_{i=1}^Nx_iy_i\right) \\ & = (X'X + \lambda I_k)^{-1}X'y \\ & = (X'X + \lambda I_k)^{-1}X'(X\beta + e)\\ & = (X'X + \lambda I_k)^{-1}X'X\beta \end{aligned} \end{equation} and hence it is unbiased only if $\lambda = 0$.

I am stuck on number 2. For $\hat\beta$ to be consistent I need $$(X'X + \lambda I_k)^{-1}X'X \xrightarrow{p} I$$ but how it could be the case for $\lambda > 0$?

$\endgroup$
2
  • $\begingroup$ Looks like ridge regression (just to note). $\endgroup$ Nov 13, 2016 at 20:01
  • $\begingroup$ @tosik, the last line of the derivation is true in expectation only. $\endgroup$ Dec 2, 2021 at 14:51

1 Answer 1

7
$\begingroup$

Note that $$ \text{plim} \Big[(X'X + \lambda I_k)^{-1} X'X\Big] =\text{plim}(n^{-1}X'X + n^{-1}\lambda I_k)^{-1}\text{plim}(n^{-1}X'X)$$

The second plim converges by asumption. For the first we have $$\text{plim}(n^{-1}X'X + n^{-1}\lambda I_k)^{-1}=\left(\text{plim}n^{-1}X'X + \text{plim}n^{-1}\lambda I_k\right)^{-1} $$

and that

$$\text{plim}n^{-1}\lambda I_k = \text{lim}n^{-1}\lambda I_k = 0$$

leading to the desired consistency result. Intuitively the purpose of adding a term like $\lambda I_k$ is to handle a "bad sample", i.e. it is a finite-sample "tactic" to get results, but whose effect is eliminated asymptotically.

$\endgroup$
2
  • $\begingroup$ I accepted because I understood what Alecos meant $\endgroup$
    – tosik
    Nov 14, 2016 at 12:43
  • $\begingroup$ @MrFrog It is not. Notice that in the first plim expression there is a ${-1}$ outside, so essentially the two $n^{-1}$ cancel out. I just added $n^{-1}/n^{-1} =1$ and then split them. $\endgroup$ Mar 25 at 20:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.