I'm reading through Fan and Li's paper "Variable Selection via Nonconcave Penalized Likelihood and its Oracle Properties". In Page 2 near bottom right corner, they proposed three properties that a good penalized estimator should have:

  1. Unbiasedness: The resulting estimator is nearly unbiased when the true unknown parameter is large to avoid unnecessary modeling bias.
  2. Sparsity: The resulting estimator is a thresholding rule, which automatically sets small estimated coefficient to zero to reduce model complexity.
  3. Continuity:The resulting estimator is continuous in data z to avoid instability in model prediction.

They then showed that the hard thresholding penalty function can result in estimators with those good properties, given some conditions satisfied. Specifically,

  • (1) A sufficient condition for unbiasedness is $p_\lambda'(\theta)=0$ for large $|z|$;
  • (2) A sufficient condition for sparsity is $\text{min}_{\theta \neq 0} \{|\theta| + p_\lambda'(\theta)\} \gt 0$;
  • (3) A sufficient and necessary condition for continuity is that the minimum of the function $|\theta| + p_\lambda'(\theta)$ is attained at 0.

My questions are:

  1. For proving condition (2), they depicted a figure below. I am confused why $\theta + p_\lambda'(\theta)$ is a Parabola since it is equal to $\theta + 2(\lambda - \theta)I(\theta < \lambda)$. And moreover, how does the proof to condition (2) come from this figure?

Update: Figure 3 should be a general illustration about $\theta + p_\lambda'(\theta)$ against $\theta (>0)$, as pointed out by @Glen_b.

  1. The last sentence of first paragraph in Page 3 concludes that "a penalty function satisfying the conditions of sparsity and continuity must be singular at the origin". Why is that?

Obviously I totally missed the points here. Could any of you walk me through the arguments made for proving conditions (2) and (3)? Thank you very much for the help.

enter image description here

  • $\begingroup$ It's possible I missed something, but I don't see anything in the paper that suggests the depicted curve is actually parabolic, nor that it specifically represents exactly the function you claim it does. You make a number of other assertions that are also not clear to me from the paper. In any case, you should justify each of your assertions of fact. How, for example do you know that the depicted figure specifically shows $\theta + 2(\lambda - \theta)I(\theta < \lambda)$, rather than it being intended more generally to represent any of a variety of possible $\theta + p_\lambda'(\theta)$? $\endgroup$ – Glen_b Jul 6 '15 at 10:56
  • $\begingroup$ @Glen_b, thanks for pointing out this. I think you're right. Figure 3 should be general case. But given this, I still have trouble to understand the conclusion "the minimum of the function $|\theta| + p_{\lambda}'(\theta)$ is positive". Could you shed light on this? $\endgroup$ – Aaron Zeng Jul 6 '15 at 15:35

For conclusion A sufficient condition for sparsity is $\text{min}_{\theta \neq 0} \{|\theta| + p_\lambda'(\theta)\} \gt 0$, I think I got some thoughts.

Given the condition (2) $\text{min}_{\theta \neq 0} \{|\theta| + p_\lambda'(\theta)\} \gt 0$, then we have two cases regarding $|z|$.

  1. If $|z| \lt \text{min}_{\theta \neq 0} \{|\theta| + p_\lambda'(\theta)\}$, then according to Figure 3, the first-order derivative is positive for $\theta > 0$, and negative for $\theta < 0$. That is, the objective function is monotonously decreasing in $(-\infty, 0)$, and increasing in $(0, +\infty)$. Therefore, the penalized estimator $\hat{\theta} = 0$ in this case. Note that the objective function could be nondifferentiable at $\theta = 0$. Note also that $|z| = |\hat{\theta}^{OLS}|$ in the orthonormal case. That means, when the least square estimates are small enough (within a range), then the penalized estimates would be shrunk to 0, which leads to sparsity.

  2. If $|z| \gt \text{min}_{\theta \neq 0} \{|\theta| + p_\lambda'(\theta)\}$, then according to Figure 3, there are two roots, say $r_1, r_2$, to the first-order equation. Thus, the objective function first increases untill $\theta = r_1$, then decreases until $\theta = r_2$, then increases again. That is, $\theta = r_2$ is the minimizer, thus the penalized estimator. Since $|z|$ is sufficiently large, we actually don't need to shrink the estimator since $\hat{\theta} = z$ is approximately unbiased if condition (1) is satisfied.

On the other hand, if the sufficient condition doesn't hold, we then cannot find a situation where the objective function can be monotonously decreasing in $(-\infty, 0)$ and increasing $(0, +\infty)$. That is, the penalized estimator will not be zero.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.