40
$\begingroup$

Least-angle regression and the lasso tend to produce very similar regularization paths (identical except when a coefficient crosses zero.)

They both can be efficiently fit by virtually identical algorithms.

Is there ever any practical reason to prefer one method over the other?

$\endgroup$
1
  • $\begingroup$ If you reassessed the answers at this point, would you choose a different "accepted" answer? $\endgroup$
    – Aaron Hall
    May 1, 2017 at 3:22

6 Answers 6

13
$\begingroup$

The "no free lunch" theorems suggest that there are no a-priori distinctions between statistical inference algorithms, i.e. whether LARS or LASSO works best depends on the nature of the particular dataset. In practice then, it is best to try both and use some reliable estimator of generalisation performance to decide which to use in operation (or use an ensemble). As the differences between LARS and LASSO are rather slight, the differences in performance are likely to be rather slight as well, but in general there is only one way to find out for sure!

$\endgroup$
1
  • $\begingroup$ Can you expand on possible 'ensemble method' in this particular case? $\endgroup$
    – chl
    Nov 22, 2010 at 22:35
41
$\begingroup$

When used in stage-wise mode, the LARS algorithm is a greedy method that does not yield a provably consistent estimator (in other words, it does not converge to a stable result when you increase the number of samples).

Conversely, the LASSO (and thus the LARS algorithm when used in LASSO mode) solves a convex data fitting problem. In particular, this problem (the L1 penalized linear estimator) has plenty of nice proved properties (consistency, sparsistency).

I would thus try to always use the LARS in LASSO mode (or use another solver for LASSO), unless you have very good reasons to prefer stage-wise.

$\endgroup$
0
11
$\begingroup$

As mentioned before, LARS is a particular method to solve the Lasso problem, i.e. the $l_1$-regularized least squares problem. Its success stems from the fact that it requires an asymptotic effort comparable to standard least-squares regression, and thus a highly superior performance than required by the solution of a quadratic programming problem. Later extensions of LARS also adressed the more general elastic-net problem where you include a sum of $l_1$ and $l_2$-regularization terms into the least-squares functional.

The intention of this answer is to point out that LARS nowadays seems to have been superseeded by coordinate-descent and stochastic coordinate-descent methods. These methods are based on particularly simple algorithms, while at the same time the performance seems to be higher than that of LARS (often one or two orders of magnitude faster). For examples see this paper of Friedman et al.

So, if you plan to implement LARS, don't. Use coordinate-descent which takes a few hours.

$\endgroup$
1
  • 1
    $\begingroup$ +1 for not implementing LARS but coordinate descent: it does have settings where it is better than coordinate descent (for instance for small and mod-size problems that are very sparse, see the thesis of Julien Mairal for empirical comparisons), but it is very hard to implement right, much harder than coordinate descent. $\endgroup$ Apr 28, 2016 at 9:49
10
$\begingroup$

LASSO is not an algorithm per se, but an operator.

There are many different ways to derive efficient algorithms for $\ell_1$ regularized problems. For instance, one can use quadratic programming to them tackle directly. I guess this is what you refer to as LASSO.

Another one is LARS, very popular because of its simplicity, connection with forward procedures (yet not too greedy), very constructive proof and easy generalization.

Even compared with state of the art quadratic programming solvers, LARS can be much more efficient.

$\endgroup$
5
$\begingroup$

The computation of the lasso solutions is a quadratic programming problem, and can be tackled by standard numerical analysis algorithms. But the least angle regression procedure is a better approach. This algorithm exploits the special structure of the lasso problem, and provides an efficient way to compute the solutions simultaneously for all values of $\lambda$.


Here is my opinion:

Your question can be divided to two parts. High dimensional cases and low dimensional cases. On the other hand it depends on what criteria are you going to use for selecting the optimal model. in the original paper of LARS, it is proved a $C_p$ criteria for selecting the best model and also you can see a SVS and CV criteria in the 'Discussion' of the paper as well. Generally, there are tiny differences between LARS and Lasso and can be ignored completely.

In addition LARS is computationally fast and reliable. Lasso is fast but there is a tiny difference between algorithm that causes the LARS win the speed challenge. On the other hand there are alternative packages for example in R, called 'glmnet' that work more reliable than lars package(because it is more general).

To sum up, there is nothing significant that can be considered about lars and lasso. It depended on the context you are going to use model.

I personally advise using glmnet in R in both high and low dimensional cases. or if you are interested in different criteria, you can use http://cran.r-project.org/web/packages/msgps/ package.

$\endgroup$
-1
$\begingroup$

In some contexts a regularized version of the least squares solution may be preferable. The LASSO (least absolute shrinkage and selection operator) algorithm, for example, finds a least-squares solution with the constraint that | β | 1, the L1-norm of the parameter vector, is no greater than a given value. Equivalently, it may solve an unconstrained minimization of the least-squares penalty with α | β | 1 added, where α is a constant (this is the Lagrangian form of the constrained problem.) This problem may be solved using quadratic programming or more general convex optimization methods, as well as by specific algorithms such as the least angle regression algorithm. The L1-regularized formulation is useful in some contexts due to its tendency to prefer solutions with fewer nonzero parameter values, effectively reducing the number of variables upon which the given solution is dependent.[11] For this reason, the LASSO and its variants are fundamental to the field of compressed sensing.

$\endgroup$
4
  • 6
    $\begingroup$ With respect, this looks like a direct copy-and-paste from Wikipedia, and doesn't really answer the question. $\endgroup$
    – NPE
    Nov 18, 2010 at 9:51
  • 4
    $\begingroup$ (-1) At the very least, you should acknowledge the quoting from Wikipedia, § on LASSO method at en.wikipedia.org/wiki/Least_squares!!! BTW you forgot to paste the 11th reference. $\endgroup$
    – chl
    Nov 18, 2010 at 10:15
  • $\begingroup$ I forgot to put the link, it is true, but anyway I think that is a good reply for this questions. Sorry if I made you think I wrote that $\endgroup$ Nov 18, 2010 at 10:25
  • $\begingroup$ It would be more helpful to refer to The Lasso Page in that case. Now, the question is about pros and cons of LAR and Lasso, not about what Lasso actually does. The LARS algorithm might easily be modified to produce solutions for other estimators, like the Lasso; it works well in the $n\ll p$ case, but it is sensitive to the effects of noise (because it is based upon an iterative refitting of the residuals), as quoted from scikit-learn.sourceforge.net/modules/glm.html. $\endgroup$
    – chl
    Nov 18, 2010 at 10:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.