# Soft-thresholding vs. Lasso penalization

I am trying to summarize what I understood so far in penalized multivariate analysis with high-dimensional data sets, and I still struggle through getting a proper definition of soft-thresholding vs. Lasso (or $L_1$) penalization.

More precisely, I used sparse PLS regression to analyze 2-block data structure including genomic data (single nucleotide polymorphisms, where we consider the frequency of the minor allele in the range {0,1,2}, considered as a numerical variable) and continuous phenotypes (scores quantifying personality traits or cerebral asymmetry, also treated as continuous variables). The idea was to isolate the most influential predictors (here, the genetic variations on the DNA sequence) to explain inter-individual phenotypic variations.

I initially used the mixOmics R package (formerly integrOmics) which features penalized PLS regression and regularized CCA. Looking at the R code, we found that the "sparsity" in the predictors is simply induced by selecting the top $k$ variables with highest loadings (in absolute value) on the $i$th component, $i=1,\dots, k$ (the algorithm is iterative and compute variables loadings on $k$ components, deflating the predictors block at each iteration, see Sparse PLS: Variable Selection when Integrating Omics data for an overview). On the contrary, the spls package co-authored by S. Keleş (see Sparse Partial Least Squares Regression for Simultaneous Dimension Reduction and Variable Selection, for a more formal description of the approach undertaken by these authors) implements $L_1$-penalization for variable penalization.

It is not obvious to me whether there is a strict "bijection", so to say, between iterative feature selection based on soft-thresholding and $L_1$ regularization. So my question is: Is there any mathematical connection between the two?

References

1. Chun, H. and Kele ̧s, S. (2010), Sparse partial least squares for simultaneous dimension reduction and variable selection. Journal of the Royal Statistical Society: Series B, 72, 3–25.
2. Le Cao, K.-A., Rossouw, D., Robert-Granie, C., and Besse, P. (2008), A Sparse PLS for Variable Selection when Integrating Omics Data. Statistical Applications in Genetics and Molecular Biology, 7, Article 35.

What i'll say holds for regression, but should be true for PLS also. So it's not a bijection because depeding on how much you enforce the constrained in the $l1$, you will have a variety of 'answers' while the second solution admits only $p$ possible answers (where $p$ is the number of variables) <-> there are more solutions in the $l1$ formulation than in the 'truncation' formulation.

• @kwak Ok, the LARS algorithm seems largely more sophisticated than simple thresholding on variable importance, but the point is that I don't see a clear relation between the penalty parameter and the # of variables that are asked to be kept in the model; it seems to me we cannot necessarily find a penalty parameter that would yield exactly a fixed # of variables.
– chl
Sep 23, 2010 at 9:51
• @chl:> S-PLS you mean ?(you wrote LARS which is a different thing from either algorithm you discuss). Indeed, there is a monotoneous relationship between the penalty parameter and the # of component, but it is not a linear relationsip and this relationship varies on a case per case basis (is dataset/problem dependant). Sep 23, 2010 at 10:04
• @kwak L1-penalty may be achieved using LARS, unless I am misleading. Your second point is what I have in mind in fact; have you any reference about that point?
– chl
Sep 23, 2010 at 10:21
• @chl:>* L1-penalty may be achieved using LARS, unless I am misleading* i didn't know that (and sort of doubt it). Can you provide a reference ? Thanks. for your second question: look On the “degrees of freedom” of the lasso Hui Zou, Trevor Hastie, and Robert Tibshirani Source: Ann. Statist. Volume 35, Number 5 (2007), 2173-2192. (there are many ungated versions). Sep 23, 2010 at 10:47
• @kwak Check out Tibshirani's webpage, www-stat.stanford.edu/~tibs/lasso.html and the lars R package; other methods include coordinate descent (see JSS 2010 33(1), bit.ly/bDNUFo), and the Python scikit.learn package features both approaches, bit.ly/bfhnZz.
– chl
Sep 23, 2010 at 10:58

$L_1$ penalization is part of an optimization problem. Soft-thresholding is part of an algorithm. Sometimes $L_1$ penalization leads to soft-thresholding.

For regression, $L_1$ penalized least squares (Lasso) results in soft-thresholding when the columns of the $X$ matrix are orthogonal (assuming the rows correspond to different samples). It is really straight-forward to derive when you consider the special case of mean estimation, where the $X$ matrix consists of a single $1$ in each row and zeroes everywhere else.

For the general $X$ matrix, computing the Lasso solution via cyclic coordinate descent results in essentially iterative soft-thresholding. See http://projecteuclid.org/euclid.aoas/1196438020 .

• (+1) Thanks for this, especially Friedman's paper.
– chl
Dec 21, 2010 at 15:30