Lp regularization with p < 1? Lasso regularization has the nice property of creating coefficient sparsity. Generalizing to Lp regularization, is it possible to regularize a logistic regression with p < 1? I would like to control the sparsity, possibly at the expense of performance. Is there a way to do this in Python? Scikit-Learn only has L1/L2.
 A: In the context of linear regression, Frank and Friedman (1993) introduced 'bridge regression'. The goal is to minimize the squared error plus a penalty on the $\ell_p$ norm of the weights.
$$\min_w \sum_{i=1}^n (y - w \cdot x_i)^2 + \lambda \|w\|_p$$
$p=2$ corresponds to ridge regression, $p=1$ to LASSO, and $p=0$ to subset selection. As discussed by Tibshirani (1996) and Fu (1998), $p \le 1$ yields sparse solutions, and smaller values of $p$ correspond to priors on the weights that are increasingly concentrated around the axes. That is, smaller values of $p$ correspond to a preference for increasingly sparse solutions. There are also similarities between bridge regression with $1 \le p \le 2$ and elastic net regression (a combination of $\ell_1$ and $\ell_2$ penalties), which I discuss here.
The penalty term is convex for $p \ge 1$, but not for $p < 1$, which makes solving the optimization problem much more difficult in the latter case. Heuristic strategies have been developed. For example, Orthogonal Matching Pursuit (Pati et al. 1993) is a greedy strategy for obtaining approximate solutions when $p=0$ (which is an NP-hard problem). However, the lasso and elastic net are currently preferred in most circumstances. They work well in practice, they have well understood properties, and efficient solvers are available.
If your goal is to control the sparsity, it would make sense to start by penalizing the $\ell_1$ norm, and adjusting the strength of the penalty term (as mentioned in the comments). This is much simpler than dealing with the nasty, nonconvex optimization problem you'd have with $p < 1$.
References
Frank and Friedman (1993). A Statistical View of Some Chemometrics Regression Tools.
Fu (1998). Penalized Regressions: The Bridge versus the Lasso.
Pati et al. (1993). Orthogonal matching pursuit: recursive function approximation with applications to wavelet decomposition.
Tibshirani (1996). Regression Shrinkage and Selection via the Lasso.
Zou and Hastie (2005). Regularization and variable selection via the elastic net.
