# Consistency of lasso

I would appreciate help in understanding the following theorem from Knight and Fu (2002) paper:

Consider linear regression model of the form $$Y_i = \beta_0 + x_i'\beta + \varepsilon_i,$$ where $\varepsilon_i \sim i.i.d.(0, \sigma^2)$. We estimate the model using the lasso type estimator $\hat\beta$ $$\hat\beta = \text{arg min}_\phi \sum_{i=1}^{n}(Y_i - x_i'\phi)^2 + \lambda_n\sum_{j=1}^p\vert\phi_j\vert .$$

Further assume that $C_n = n^{-1}\sum_{i=1}^nx_ix_i' \to C$, where $C$ is nonsingular. Let $\lambda_n/n \to \lambda_0 \geq 0$. Then $\hat\beta \xrightarrow{p} \text{arg min}(Z)$ where $$Z(\phi) = (\phi - \beta)'C(\phi - \beta) + \lambda_0\sum_{j=1}^p\vert \phi_j\vert.$$

Thus if $\lambda_n = o(n)$, $argmin(Z) = \beta$ and so $\hat\beta$ is consistent.

Could you explain how does consistency follow from the fact that $\hat\beta \xrightarrow{p} \text{arg min}(Z)$. And also do we talk about consistency in parameter estimation or other types of consistency (e.g. oracle properties)?

• It’s simply that if it’s $o(n)$, $\lambda_0=0$ and so asymptotically you have just the quadratic form to minimize. Intuitively, it means that you have a penalized estimator for which the penalization is loosened with $n$. – hejseb May 26 '18 at 18:39
• @hejseb thank you, this is helpful. If you could also explain how $\sum_{i=1}^n(Y_i - x_i'\phi)^2$ converges to $(\phi - \beta)'C(\phi - \beta)$ and post everything as an answer, I would accept it. – tosik May 27 '18 at 9:19
• If you write the quadratic as $(Y-X\phi)'(Y-X\phi)$ you get (using $Y=X\beta+\epsilon$) that it's $(\beta-\phi)'X'X(\beta-\phi)$ plus terms which disappear if $\epsilon$ and $X$ are independent. Divide by $n$ and then using $X'X/n\to C$ you get it. I'll write up an answer in a few days. – hejseb May 29 '18 at 6:24