1
$\begingroup$

In the Knight and Fu's paper, in Equation 6 authors consider the pointwise convergence in probability as

$$\underset{\phi \in K}{\operatorname{sup}} | Z_n(\phi)-Z(\phi)-\sigma^2| \longrightarrow_p 0$$

Why is there extra $\sigma^2$ term? How did authors derive the term? How is it different from classical pointwise convergence $$\underset{\phi \in K}{\operatorname{sup}} | Z_n(\phi)-Z(\phi)| \longrightarrow_p 0$$

Can anyone point me to the proof or theory related to this?

Keith Knight and Wenjiang Fu. "Asymptotics for Lasso-Type Estimators." The Annals of Statistics, Vol. 28, No. 5 (Oct., 2000), pp. 1356-1378.

$\endgroup$
  • 1
    $\begingroup$ I've added a citation to your post as a check against link rot. I also capitalized the authors' names to conform to conventional English, and removed the boilerplate "thanks" to conform to CV style. $\endgroup$ – Sycorax says Reinstate Monica Dec 7 '14 at 21:58
2
$\begingroup$

Pointwise, $Z_n(\phi) \to Z(\phi) + \sigma^2$ because \begin{align*} Z_n(\phi) &= \frac 1 n (Y - X\phi)^T(Y - X\phi) + \frac {\lambda_n} n \sum_p |\phi_p|^\gamma \\ &= \underbrace{\frac 1 n (Y - X \hat \beta)^T(Y - X\hat \beta)}_{\to \sigma^2} \\ & \qquad + \underbrace{\frac 1 n (X\hat \beta - X\phi)^T(X\hat \beta - X\phi) + \frac{\lambda_n}{n} \sum_p |\phi_p|^\gamma}_{\to Z(\phi)}. \end{align*} So, the point of the proof is that this convergence is uniform on compact sets rather than just holding pointwise. Note that $$ \frac 1 n (X\hat \beta - X\phi)^T(X\hat \beta - X\phi) = (\hat \beta - \phi) ^T\left[\frac 1 n X^TX\right](\hat \beta - \phi) $$ with $\frac 1 n X^T X \to C$ and $\hat \beta \to \beta$ (all convergence is in probability, and I'm being a little sloppy in ignoring dependence of some things on $n$).

$\endgroup$
  • $\begingroup$ In your derivation, $\hat{\beta}$ seems to me is the least squares estimator, that's why the crossproduct term cancels out. However, $\beta$ defined in $Z(\phi)$ is the minimizer to the penalized objective. So I don't understand how they match up, and how your second part converges to $Z(\phi)$. Could you explain this to me? $\endgroup$ – Aaron Zeng Jul 8 '15 at 14:50
  • 1
    $\begingroup$ @AaronZeng The $\beta$ defined in $Z(\phi)$ is the true vector of regression coefficients; I'm using $\hat \beta$ here for the LS-estimator. In the notation of the paper, this is would be written $\hat \beta^{(0)}_n$. As noted on the bottom of p1357, it is "well-known" that $\hat \beta^{(0)}_n \to \beta$ in probability (in fact, it is $\sqrt n$ consistent). $\endgroup$ – guy Jul 8 '15 at 19:30
0
$\begingroup$

The $\sigma^2$ term is introduced at the bottom of p1357, as the dispersion parameter of the asymptotic variance of the estimator $\hat \beta^{(0)}_n$. Equation 6 is a sufficient condition for $Z_n(\phi)$ to converge pointwise to $Z(\phi) + \sigma^2$.

$\endgroup$
  • $\begingroup$ I am sorry, can you elaborate! I understood that $\sigma^2$ is the dispersion parameter but how did they derive $$\underset{\phi \in K}{\operatorname{sup}} | Z_n(\phi)-Z(\phi)-\sigma^2| \longrightarrow_p 0$$ $\endgroup$ – user52705 Dec 7 '14 at 2:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.