9
$\begingroup$

I've seen the terms LASSO regularisation and LASSO penalisation used interchangeably? Is this correct, are they the same thing or what are the differences?

Thanks!

$\endgroup$
18
$\begingroup$

In mathematics, statistics and physics, regularisation is the process of adding information in order to make an ill-posed problem soluble and well-behaved or to force a problem to exhibit some property known to be satisfied by suitable solutions; it particularly applies to objective functions in ill-posed optimisation problems.

With regard to the lasso, the added information pertains to the $\ell_1-$norm of some parameter vector and, due to convexity and the geometry of the extreme points of a polyhedron specified by a constraint of the form $$ \sum_{j=1}^{p} \lvert \beta_j \rvert \leq t,$$ with $t>0$ large enough, this comes down to specifying that some of the parameters $\beta_j$ vanish.

Whilst having the same meaning, the term penalisation is, to my knowledge, mostly used by statisticians and data scientists ; it has the merit of highlighting the fact that one regularises a problem by penalising solutions that stray from a certain desirable behaviour (e.g. sparsity).

$\endgroup$
  • 1
    $\begingroup$ Ah so that's what regularisation means. Thank you. $\endgroup$ – Ingolifs Dec 18 '19 at 23:00
  • $\begingroup$ I do not think Lasso as such specifies sparsity i.e. that some of the parameters vanish; instead this is common effect of the $\mathcal{l}_1$-norm which is not shared with the $\mathcal{l}_2$-norm $\endgroup$ – Henry Dec 19 '19 at 14:04
  • 2
    $\begingroup$ @Henry I don't think the answer contradicts this in any way. Effectively if we deem "sparse" solutions to be more "regular" in the sense described above, then a 'penaliser' relying on the l1 norm is an effective 'regularizer' (i.e. it is an effective means of achieving more 'regular' solutions), w.r.t. this particular definition of what it means to be 'regular'. $\endgroup$ – Tasos Papastylianou Dec 19 '19 at 15:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.