How to prove the properties of penalized likelihood estimator in Fan and Li (2001) paper

Question

I'm reading through Fan and Li (2001) Variable Selection via Nonconcave Penalized Likelihood and its Oracle Properties. On p. 1349 (near the bottom-right corner) they proposed three properties that a good penalized estimator should have:

1. Unbiasedness: The resulting estimator is nearly unbiased when the true unknown parameter is large to avoid unnecessary modeling bias.
2. Sparsity: The resulting estimator is a thresholding rule, which automatically sets small estimated coefficient to zero to reduce model complexity.
3. Continuity:The resulting estimator is continuous in data z to avoid instability in model prediction.

They then showed that the hard thresholding penalty function can result in estimators with those good properties, given some conditions satisfied. Specifically,

• (1) A sufficient condition for unbiasedness is $p_\lambda'(\theta)=0$ for large $|z|$;
• (2) A sufficient condition for sparsity is $\text{min}_{\theta \neq 0} \{|\theta| + p_\lambda'(\theta)\} \gt 0$;
• (3) A sufficient and necessary condition for continuity is that the minimum of the function $|\theta| + p_\lambda'(\theta)$ is attained at 0.

My questions are:

1. For proving condition (2), they depicted a figure below. I am confused why $\theta + p_\lambda'(\theta)$ is a Parabola since it is equal to $\theta + 2(\lambda - \theta)I(\theta < \lambda)$. And moreover, how does the proof to condition (2) come from this figure?

Update: Figure 3 should be a general illustration about $\theta + p_\lambda'(\theta)$ against $\theta (>0)$, as pointed out by @Glen_b.

2. The last sentence of first paragraph in Page 3 concludes that "a penalty function satisfying the conditions of sparsity and continuity must be singular at the origin". Why is that?

Obviously I totally missed the points here. Could any of you walk me through the arguments made for proving conditions (2) and (3)? Thank you very much for the help.

Answer 1

A sufficient condition for sparsity is that $\min_{\theta \neq 0} [ |\theta| + p'_\lambda (|\theta|) ]$ is positive.

The reason for this condition is explained on p. 1350 of the paper, but it is only a brief outline, as the authors have chosen to omit most of the demonstration of the asserted results. I will try to fill in the blanks of what they have omitted.

---

The mathematical problem being solved here is to find a value $\hat{\theta} > 0$ that minimises a particular objective function for a fixed value of $z > 0$. The objective function specified in equation $(2.3)$ is:

$$\begin{equation} \begin{aligned} F(z,\theta) &\equiv \tfrac{1}{2} (z-\theta)^2 + p_\lambda(|\theta|) \\[6pt] &= \tfrac{1}{2} z^2 - z \theta + \tfrac{1}{2} \theta^2 + p_\lambda(|\theta|) \\[6pt] &= \tfrac{1}{2} \theta^2 - z \theta + p_\lambda(|\theta|) + \text{const.} \\[6pt] \end{aligned} \end{equation}$$

Finding the minimising argument is done by ordinary calculus techniques. To facilitate this analysis, we define the function $H_\lambda(\theta) \equiv |\theta| + p'_\lambda (|\theta|)$. Using the chain rule, for all $\theta \neq 0$ the first derivative of this objective function is:

$$\begin{equation} \begin{aligned} \quad \text{ } \text{ } F'(z,\theta) &= \theta + \text{sgn}(\theta) \cdot p'_\lambda (|\theta|) -z \\[6pt] &= \text{sgn}(\theta) \cdot |\theta| + \text{sgn}(\theta) \cdot p'_\lambda (|\theta|) -z \\[6pt] &= \text{sgn}(\theta) \Big[ |\theta| + p'_\lambda (|\theta|) \Big] -z \\[6pt] &= \text{sgn}(\theta) \cdot H_\lambda(\theta) -z. \\[6pt] \end{aligned} \end{equation}$$

This gets us to the point where the derivative of the objective function depends on the function $H_\lambda(\theta)$, which is the function subject to the condition in question. To complete the examination of the minimisation problem, the authors look at what happens in different cases when the minimum of this function is positive, negative, or zero. This discussion occurs on p. 1350 of the paper, and with the above form for the derivative of the objective function, it should now be easier to understand.

(As some other commentators have also noted, Figure 3 appears to me to be intended as a generic figure, rather than one corresponding to a specific form of probability function.)

Answer 2

For conclusion A sufficient condition for sparsity is $\text{min}_{\theta \neq 0} \{|\theta| + p_\lambda'(\theta)\} \gt 0$, I think I got some thoughts.

Given the condition (2) $\text{min}_{\theta \neq 0} \{|\theta| + p_\lambda'(\theta)\} \gt 0$, then we have two cases regarding $|z|$.

1. If $|z| \lt \text{min}_{\theta \neq 0} \{|\theta| + p_\lambda'(\theta)\}$, then according to Figure 3, the first-order derivative is positive for $\theta > 0$, and negative for $\theta < 0$. That is, the objective function is monotonously decreasing in $(-\infty, 0)$, and increasing in $(0, +\infty)$. Therefore, the penalized estimator $\hat{\theta} = 0$ in this case. Note that the objective function could be nondifferentiable at $\theta = 0$. Note also that $|z| = |\hat{\theta}^{OLS}|$ in the orthonormal case. That means, when the least square estimates are small enough (within a range), then the penalized estimates would be shrunk to 0, which leads to sparsity.

2. If $|z| \gt \text{min}_{\theta \neq 0} \{|\theta| + p_\lambda'(\theta)\}$, then according to Figure 3, there are two roots, say $r_1, r_2$, to the first-order equation. Thus, the objective function first increases untill $\theta = r_1$, then decreases until $\theta = r_2$, then increases again. That is, $\theta = r_2$ is the minimizer, thus the penalized estimator. Since $|z|$ is sufficiently large, we actually don't need to shrink the estimator since $\hat{\theta} = z$ is approximately unbiased if condition (1) is satisfied.

On the other hand, if the sufficient condition doesn't hold, we then cannot find a situation where the objective function can be monotonously decreasing in $(-\infty, 0)$ and increasing $(0, +\infty)$. That is, the penalized estimator will not be zero.