# Pointwise convergence in probablity of lasso

In the Knight and Fu's paper, in Equation 6 authors consider the pointwise convergence in probability as

$$\underset{\phi \in K}{\operatorname{sup}} | Z_n(\phi)-Z(\phi)-\sigma^2| \longrightarrow_p 0$$

Why is there extra $\sigma^2$ term? How did authors derive the term? How is it different from classical pointwise convergence $$\underset{\phi \in K}{\operatorname{sup}} | Z_n(\phi)-Z(\phi)| \longrightarrow_p 0$$

Can anyone point me to the proof or theory related to this?

Keith Knight and Wenjiang Fu. "Asymptotics for Lasso-Type Estimators." The Annals of Statistics, Vol. 28, No. 5 (Oct., 2000), pp. 1356-1378.

• I've added a citation to your post as a check against link rot. I also capitalized the authors' names to conform to conventional English, and removed the boilerplate "thanks" to conform to CV style. – Sycorax says Reinstate Monica Dec 7 '14 at 21:58

Pointwise, $Z_n(\phi) \to Z(\phi) + \sigma^2$ because \begin{align*} Z_n(\phi) &= \frac 1 n (Y - X\phi)^T(Y - X\phi) + \frac {\lambda_n} n \sum_p |\phi_p|^\gamma \\ &= \underbrace{\frac 1 n (Y - X \hat \beta)^T(Y - X\hat \beta)}_{\to \sigma^2} \\ & \qquad + \underbrace{\frac 1 n (X\hat \beta - X\phi)^T(X\hat \beta - X\phi) + \frac{\lambda_n}{n} \sum_p |\phi_p|^\gamma}_{\to Z(\phi)}. \end{align*} So, the point of the proof is that this convergence is uniform on compact sets rather than just holding pointwise. Note that $$\frac 1 n (X\hat \beta - X\phi)^T(X\hat \beta - X\phi) = (\hat \beta - \phi) ^T\left[\frac 1 n X^TX\right](\hat \beta - \phi)$$ with $\frac 1 n X^T X \to C$ and $\hat \beta \to \beta$ (all convergence is in probability, and I'm being a little sloppy in ignoring dependence of some things on $n$).
• In your derivation, $\hat{\beta}$ seems to me is the least squares estimator, that's why the crossproduct term cancels out. However, $\beta$ defined in $Z(\phi)$ is the minimizer to the penalized objective. So I don't understand how they match up, and how your second part converges to $Z(\phi)$. Could you explain this to me? – Aaron Zeng Jul 8 '15 at 14:50
• @AaronZeng The $\beta$ defined in $Z(\phi)$ is the true vector of regression coefficients; I'm using $\hat \beta$ here for the LS-estimator. In the notation of the paper, this is would be written $\hat \beta^{(0)}_n$. As noted on the bottom of p1357, it is "well-known" that $\hat \beta^{(0)}_n \to \beta$ in probability (in fact, it is $\sqrt n$ consistent). – guy Jul 8 '15 at 19:30
The $\sigma^2$ term is introduced at the bottom of p1357, as the dispersion parameter of the asymptotic variance of the estimator $\hat \beta^{(0)}_n$. Equation 6 is a sufficient condition for $Z_n(\phi)$ to converge pointwise to $Z(\phi) + \sigma^2$.
• I am sorry, can you elaborate! I understood that $\sigma^2$ is the dispersion parameter but how did they derive $$\underset{\phi \in K}{\operatorname{sup}} | Z_n(\phi)-Z(\phi)-\sigma^2| \longrightarrow_p 0$$ – user52705 Dec 7 '14 at 2:47