# Prove consistency

Consider the following estimator $$\hat\beta = \left(\sum_{i=1}^{N}x_ix_i' + \lambda I_k\right)^{-1}\left(\sum_{i=1}^Nx_iy_i\right)$$ where $$x_i$$ is a column vector $$k\times1$$ from $$X$$ and $$\lambda > 0$$ is a scalar and $$\mathbb{E}(x_ie_i) = 0$$ .

1. Define bias and show that $$\hat\beta$$ is biased
2. Define consistency and show that $$\hat \beta$$ is consistent
3. Define conditional variance of $$\hat\beta$$.

For number 1 I have \begin{equation} \begin{aligned} \hat\beta &= \left(\sum_{i=1}^{N}x_ix_i' + \lambda I_k\right)^{-1}\left(\sum_{i=1}^Nx_iy_i\right) \\ & = (X'X + \lambda I_k)^{-1}X'y \\ & = (X'X + \lambda I_k)^{-1}X'(X\beta + e)\\ & = (X'X + \lambda I_k)^{-1}X'X\beta \end{aligned} \end{equation} and hence it is unbiased only if $$\lambda = 0$$.

I am stuck on number 2. For $$\hat\beta$$ to be consistent I need $$(X'X + \lambda I_k)^{-1}X'X \xrightarrow{p} I$$ but how it could be the case for $$\lambda > 0$$?

• Looks like ridge regression (just to note). Nov 13, 2016 at 20:01
• @tosik, the last line of the derivation is true in expectation only. Dec 2, 2021 at 14:51

Note that $$\text{plim} \Big[(X'X + \lambda I_k)^{-1} X'X\Big] =\text{plim}(n^{-1}X'X + n^{-1}\lambda I_k)^{-1}\text{plim}(n^{-1}X'X)$$
The second plim converges by asumption. For the first we have $$\text{plim}(n^{-1}X'X + n^{-1}\lambda I_k)^{-1}=\left(\text{plim}n^{-1}X'X + \text{plim}n^{-1}\lambda I_k\right)^{-1}$$
$$\text{plim}n^{-1}\lambda I_k = \text{lim}n^{-1}\lambda I_k = 0$$
leading to the desired consistency result. Intuitively the purpose of adding a term like $\lambda I_k$ is to handle a "bad sample", i.e. it is a finite-sample "tactic" to get results, but whose effect is eliminated asymptotically.
• @MrFrog It is not. Notice that in the first plim expression there is a ${-1}$ outside, so essentially the two $n^{-1}$ cancel out. I just added $n^{-1}/n^{-1} =1$ and then split them. Mar 25, 2022 at 20:15