1
$\begingroup$

I have started learning about lasso from this review ("Least Squares Optimization with L1-Norm Regularization" by Mark Schmitd).

The optimization problem is:

$$\min_{\beta } \left\{ \frac{1}{N} \sum_{i=1}^N (y_i - x_i^T \beta)^2 \right\} \text{ subject to } \sum_{j=1}^p |\beta_j| \leq t$$

where $t$ controls the amount of regularization. There is another way of writing this problem using Lagrange multipliers:

$$\min_{ \beta \in \mathbb{R}^p } \left\{ \frac{1}{N} \left\| y - X \beta \right\|_2^2 + \lambda \| \beta \|_1 \right\}$$

Question1: How does $t$ control the amount of regularization? If $\lambda$ is large, then will more coefficients be forced to zero? If so, then how many coefficients would be zero, since LASSO does not assume a particular active set?

I used lasso in Matlab and, based on the documentation, one should choose $\lambda$ to minimize the mean squared error. Say $\lambda = 0.8$, how and where is this value used? Does large $\lambda$ imply more sparsity? What is the meaning of this term?

Question2: Does $\lambda$ lie in a range, say 0 to 1?

Improve this question
$\endgroup$
4
$\begingroup$

How does $t$ control the amount of regularization?

Reducing $t$ (or equivalently, increasing $\lambda$) constrains the $\ell_1$ norm of the coefficients to smaller values. This forces more coefficients to zero, yielding sparser solutions (a sparse vector is one that contains many zeros). It also decreases the magnitude of the nonzero coefficients (this effect is called shrinkage).

If so, then how many coefficients would be zero

Unfortunately, this depends on the problem and we can't generally know a priori. One simply has to solve the problem with different values of $\lambda$ or $t$. Say we want to select the regularization parameter that gives a particular level of sparsity (say $k$ non-zero coefficients). A computationally efficient approach is to solve the problem using the least angle regression (LARS) algorithm, which computes the full lasso solution path. This returns many choices of coefficients (one for each value of $\lambda$ or $t$). We can then select from this set the coefficients that contain $k$ non-zeros.

one should choose $\lambda$ to minimize the mean squared error

Care is needed here. $\lambda$ can be chosen to minimize squared error, but only using cross validation or an independent validation set.

Does $\lambda$ lie in a range, say 0 to 1?

$\lambda$ can take any non-negative value

Improve this answer
$\endgroup$
  • 1
    $\begingroup$ Thank you for your answer and explanation. Some terms such as "$t$ forces more coefficients" to zero is unclear to me.How does the actual numeric value of $t$ or $\lambda$ work towards making the coefficient to zero? A simple example will be helpful. If $\lambda$ = $0.81$ value, then does this mean that the magnitude of coefficient which is less than $0.81$ will become zero? $\endgroup$ – Ria George Sep 24 '17 at 0:42
  • $\begingroup$ No, it doesn't work like that. You have to consider the optimization problem. Increasing $\lambda$ means the objective function will return larger values for coefficients with larger $\ell_1$ norm. So, all else, equal, coefficients with smaller $\ell_1$ norm will be favored. And, these will tend to be sparser. $\endgroup$ – user20160 Sep 24 '17 at 2:12
  • $\begingroup$ Loosely, you could say that the squared error term wants to pull the coefficients toward values that minimize the squared error. The penalty term wants to pull each coefficient toward zero, and $\lambda$ controls the strength of this pull. The optimal coefficients are the outcome of a tradeoff between these two terms, which is governed by $\lambda$. For more info, you can find many answers on this site (and elsewhere) about why lasso produces sparse solutions. $\endgroup$ – user20160 Sep 24 '17 at 2:14
  • $\begingroup$ I see, so the role of $\lambda$ is similar or different to soft-thresholding as mentioned in the wikilink en.wikipedia.org/wiki/Lasso_(statistics) $\hat{\beta}_j = S_{N \lambda}( \hat{\beta}^\text{OLS}_j ) = \hat{\beta}^\text{OLS}_j \max \left( 0, 1 - \frac{ N \lambda }{ |\hat{\beta}^\text{OLS}_j| } \right) \text{ where } \hat{\beta}^\text{OLS}$ is the least square estimator of the coefficients. $\endgroup$ – Ria George Sep 24 '17 at 2:42
  • 1
    $\begingroup$ LARS, ADMM and coordinate descent are all iterative methods. We use them because a closed form solution is only possible in the case of orthonormal regressors. $\endgroup$ – user20160 Sep 25 '17 at 0:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.