Literature on $\ell_q$ LASSO, $q < 1$

Question

I am not sure how is $\ell_q$-LASSO called, but here I am talking about LASSO regression, with $\| \beta \|_{\ell_q}$ regularization, $q< 1$. In popular literature, such as Elements of Statistical Learning or Statistical Learning with Sparsity by Hastie et. al., I am able to find only their rough definitions with some graphs, and comments about how their non-convexity makes estimating it troublesome and computationally inefficient.

However, I am interested in a deeper discussion of the approach, such as its Oracle Properties, asymptotic behaviour regarding variable selection and parameter estimation, and various cases when the method doesn't work. So far I've tested simple cases by hand in Monte Carlo experiments and find that it is able to outperform $\ell_1$ LASSO in various cases by a significant margin, hence I assume there should be strong properties at work there. That is, I'd like to understand why it works well for me.

My questions:

1. Does this method have an official name for easier literature search?

2. If you have any links to papers with proved properties, they would be very useful for further read.

3. I am also very interested in counterexamples, that would be able to break the method. (Can't construct them by myself, as I wasn't able to find assumptions, under which, say, $\ell_q$-lasso would guarantee consistent variable selection). For example, with $\ell_1$-lasso, it is known that significant multicorrelation can easily break consistent variable selection. At the same time, I find $\ell_q$-lasso being able to deal with same examples without problems.

Note: I've mentioned variable selection in the questions only as an example, other properties are also of interest.

Answer 1

Frank & Friedman (1993) suggested the idea of bridge estimates, with penalty function $P_B=\lambda\sum_j|\alpha_j|^\gamma$, as a paradigm for understanding subset selection and ridge regression. The $\ell_0$-norm corresponds to subset selection methods, $\ell_1$ is the LASSO, and $\ell_2$ is ridge regression. They noted that it would be beneficial to estimate the parameters $\lambda$ and $\gamma$ simultaneously to widen the choice of possible models but did not develop the method any further. The $\lambda$ parameter controls the size of estimates ($\hat\alpha_j^B$) or the amount of shrinkage, while the $\gamma$ parameter determines the directions in which the parameters are aligned with respect to the coordinate axes.

When $\gamma\in(0,1)$:

• The penalty function $P_B=\lambda\sum_j|\alpha_j|^\gamma$ is concave. The figure below shows the concave penalty functions (dotted) in comparison with the LASSO penalty function (solid).

• Some parameters are set to zero and the shrinkage is inversely proportional to the size of the parameters. The figure shows the thresholding function $\hat\alpha_j - sign(\hat\alpha_j)\lambda\gamma|\hat\alpha_j|^{\gamma-1}$, where $\hat\alpha_j$ are the OLS estimates. Here, with $\lambda=4$ and $\gamma=0.25$ or $\gamma=0.5$, large parameters remain fairly untouched by the shrinkage. With the LASSO (solid line), the shrinkage is constant.

• Estimates are likely to occur on the axes. The figure shows norm balls in $\mathbb{R}^2$ (left) and $\mathbb{R}^3$ (right) for $\gamma=0.5$.

Please see pages 118-119 and 126-127 of Kirkland (2014) for a comparison of these figures with other values of $\gamma$. This masters thesis also provides an overview of other shrinkage methods.

Knight & Fu (2000) showed that bridge estimates are consistent and have asymptotic normal distributions.

The main idea behind the concave penalty functions is that large parameters are penalized less so that the resulting estimates are nearly unbiased. I know of 2 other shrinkage methods which make use of concave penalties and may be of interest to you:

• Fan & Li (2001) proposed SCAD, which was the first shrinkage method having the oracle property. Although the adaptive LASSO is oracle, the bias may decrease at a faster rate with SCAD.

• Zhang (2010) proposed MCP, which follows a similar approach to SCAD but penalizes smaller parameters less.

Despite having concave penalties which are also non-differentiable at zero, they both provide efficient algorithms for computing the solution, even in high dimensional settings when $p\geq n$.