1
$\begingroup$

In the Knight and Fu's paper, in Equation 6 authors consider the pointwise convergence in probability as

$$\underset{\phi \in K}{\operatorname{sup}} | Z_n(\phi)-Z(\phi)-\sigma^2| \longrightarrow_p 0$$

Why is there extra $\sigma^2$ term? How did authors derive the term? How is it different from classical pointwise convergence $$\underset{\phi \in K}{\operatorname{sup}} | Z_n(\phi)-Z(\phi)| \longrightarrow_p 0$$

Can anyone point me to the proof or theory related to this?

Keith Knight and Wenjiang Fu. "Asymptotics for Lasso-Type Estimators." The Annals of Statistics, Vol. 28, No. 5 (Oct., 2000), pp. 1356-1378.

$\endgroup$
1
  • 1
    $\begingroup$ I've added a citation to your post as a check against link rot. I also capitalized the authors' names to conform to conventional English, and removed the boilerplate "thanks" to conform to CV style. $\endgroup$
    – Sycorax
    Commented Dec 7, 2014 at 21:58

2 Answers 2

2
$\begingroup$

Pointwise, $Z_n(\phi) \to Z(\phi) + \sigma^2$ because \begin{align*} Z_n(\phi) &= \frac 1 n (Y - X\phi)^T(Y - X\phi) + \frac {\lambda_n} n \sum_p |\phi_p|^\gamma \\ &= \underbrace{\frac 1 n (Y - X \hat \beta)^T(Y - X\hat \beta)}_{\to \sigma^2} \\ & \qquad + \underbrace{\frac 1 n (X\hat \beta - X\phi)^T(X\hat \beta - X\phi) + \frac{\lambda_n}{n} \sum_p |\phi_p|^\gamma}_{\to Z(\phi)}. \end{align*} So, the point of the proof is that this convergence is uniform on compact sets rather than just holding pointwise. Note that $$ \frac 1 n (X\hat \beta - X\phi)^T(X\hat \beta - X\phi) = (\hat \beta - \phi) ^T\left[\frac 1 n X^TX\right](\hat \beta - \phi) $$ with $\frac 1 n X^T X \to C$ and $\hat \beta \to \beta$ (all convergence is in probability, and I'm being a little sloppy in ignoring dependence of some things on $n$).

$\endgroup$
4
  • $\begingroup$ In your derivation, $\hat{\beta}$ seems to me is the least squares estimator, that's why the crossproduct term cancels out. However, $\beta$ defined in $Z(\phi)$ is the minimizer to the penalized objective. So I don't understand how they match up, and how your second part converges to $Z(\phi)$. Could you explain this to me? $\endgroup$
    – SixSigma
    Commented Jul 8, 2015 at 14:50
  • 1
    $\begingroup$ @AaronZeng The $\beta$ defined in $Z(\phi)$ is the true vector of regression coefficients; I'm using $\hat \beta$ here for the LS-estimator. In the notation of the paper, this is would be written $\hat \beta^{(0)}_n$. As noted on the bottom of p1357, it is "well-known" that $\hat \beta^{(0)}_n \to \beta$ in probability (in fact, it is $\sqrt n$ consistent). $\endgroup$
    – guy
    Commented Jul 8, 2015 at 19:30
  • 1
    $\begingroup$ @guy In the same paper, the authors claimed that $\hat{\beta} = O_p(1)$ was the direct result of this pointwise convergence. Can you give an explanation or a reference about this, please? $\endgroup$
    – mert
    Commented May 9, 2021 at 12:53
  • $\begingroup$ @mert this means that $\hat{\beta}$ is stochastically bounded, see for example bookdown.org/ts_robinson1994/… $\endgroup$ Commented Oct 5, 2022 at 19:02
0
$\begingroup$

The $\sigma^2$ term is introduced at the bottom of p1357, as the dispersion parameter of the asymptotic variance of the estimator $\hat \beta^{(0)}_n$. Equation 6 is a sufficient condition for $Z_n(\phi)$ to converge pointwise to $Z(\phi) + \sigma^2$.

$\endgroup$
1
  • $\begingroup$ I am sorry, can you elaborate! I understood that $\sigma^2$ is the dispersion parameter but how did they derive $$\underset{\phi \in K}{\operatorname{sup}} | Z_n(\phi)-Z(\phi)-\sigma^2| \longrightarrow_p 0$$ $\endgroup$
    – user52705
    Commented Dec 7, 2014 at 2:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.