Consider the following estimator $$\hat\beta = \left(\sum_{i=1}^{N}x_ix_i' + \lambda I_k\right)^{-1}\left(\sum_{i+1}^Nx_iy_i\right)$$ where $x_i$ is a column vector $k\times1$ from $X$ and $\lambda > 0$ is a scalar and $\mathbb{E}(x_ie_i) = 0$ .

  1. Define bias and show that $\hat\beta$ is biased
  2. Define consistency and show that $\hat \beta$ is consistent
  3. Define conditional variance of $\hat\beta$.

For number 1 I have \begin{equation} \begin{aligned} \hat\beta &= \left(\sum_{i=1}^{N}x_ix_i' + \lambda I_k\right)^{-1}\left(\sum_{i+1}^Nx_iy_i\right) \\ & = (X'X + \lambda I_k)^{-1}X'y \\ & = (X'X + \lambda I_k)^{-1}X'(X\beta + e)\\ & = (X'X + \lambda I_k)^{-1}X'X\beta \end{aligned} \end{equation} and hence it is unbiased only if $\lambda = 0$.

I am stuck on number 2. For $\hat\beta$ to be consistent I need $$(X'X + \lambda I_k)^{-1}X'X \xrightarrow{p} I$$ but how it could be the case for $\lambda > 0$?

  • $\begingroup$ Looks like ridge regression (just to note). $\endgroup$ Nov 13 '16 at 20:01

Note that $$ \text{plim} \Big[(X'X + \lambda I_k)^{-1} X'X\Big] =\text{plim}(n^{-1}X'X + n^{-1}\lambda I_k)^{-1}\text{plim}(n^{-1}X'X)$$

The second plim converges by asumption. For the first we have $$\text{plim}(n^{-1}X'X + n^{-1}\lambda I_k)^{-1}=\left(\text{plim}n^{-1}X'X + \text{plim}n^{-1}\lambda I_k\right)^{-1} $$

and that

$$\text{plim}n^{-1}\lambda I_k = \text{lim}n^{-1}\lambda I_k = 0$$

leading to the desired consistency result. Intuitively the purpose of adding a term like $\lambda I_k$ is to handle a "bad sample", i.e. it is a finite-sample "tactic" to get results, but whose effect is eliminated asymptotically.

  • $\begingroup$ I accepted because I understood what Alecos meant $\endgroup$
    – tosik
    Nov 14 '16 at 12:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.