3
$\begingroup$

We can regularize a linear model with L1 or L2 regularization.

But we usually write L2 with a square: $\|x\|_2^2$ and L1 with $\|x\|_1$. It seems a little bit strange and inconsistent for me, because one has a square and another does not have.

Is the reasons that adding L2 with square can directly associate with the radius $r$?

EDIT: I was trying to ask convention but not why there is a square. I think Mark answers my question perfectly. Thanks

$\endgroup$
  • 2
    $\begingroup$ Possible duplicate of Ridge & LASSO norms $\endgroup$ – Firebug Aug 17 '16 at 14:36
  • $\begingroup$ Interesting question. I myself made the same question last week and discovered it had already been answered. This can be generalized to $p$-norm regularization as well as seen in the answers in the linked question. $\endgroup$ – Firebug Aug 17 '16 at 14:36
  • 3
    $\begingroup$ In light of the accepted answer, it doesn't seem to me that this is a duplicate. $\endgroup$ – gung Aug 17 '16 at 15:57
7
$\begingroup$

$\|x\|_2^2$ and $\|x\|_1$ are $L_p^p$ for p = 2 and 1 respectively, so they are consistent in that sense.

Nevertheless, there's no really standard convention on whether or not an L2 regularization term should be squared. In fact, there's not even standardization on whether the "baseline" portion of a "least squares" objective should be squared. You could have { $norm(Ax-b)$ or $norm(Ax-b)^2$ } plus a regularization term. In the absence of the regularization term, it doesn't matter, but with a regularization term, it does. (Squaring the norm is the "standard" way of doing it, e.g., least squares). So squaring or not the baseline portion of the objective and the regularization term (even if based on an $L_2$ norm) provides 4 different possible ways in effect of doing regularization, and although similar, they are all different.

And of course, we can change from the 2 norm to the 1 norm or something else in the baseline objective term $norm(Ax-b)$. The infinity norm would give a minimax solution, by minimizing the maximum absolute deviation of elements of $Ax-b$.

Bottom line: regularization tends to be somewhat of an ad hoc or arbitrary process, and there's more than one way to do it.

$\endgroup$
  • $\begingroup$ In terms of "pretty" notation, the exponent on the norm term prevents an extra 1/p exponent when it is expanded as a sum. This may have influenced the conventions? (i.e. sum of squares may have preceded the vector-norm notation?) $\endgroup$ – GeoMatt22 Aug 23 '16 at 19:43
  • $\begingroup$ @GeoMatt22 regarding your argument about pretty notation, I think that's fair, including the point I raised on my answer regarding the derivative, towards facilitation of gradient based optimization implementations. $\endgroup$ – Firebug Oct 13 '16 at 13:18
  • 1
    $\begingroup$ @Firebug your point is probably what I was thinking of even more (I gave +1 as soon as I saw it). $\endgroup$ – GeoMatt22 Oct 13 '16 at 13:40
4
$\begingroup$

Complementing the other answers, after reading this post, it became clear to me that the convention is so thanks to the derivative of the norm.

Given the $l_2$-norm:

$$\|x\|_2=\sqrt{\sum_{i=0}^{n}x_i^2}$$

The derivative of the $l_2$-norm depends on all the elements:

$$\frac{d\|x\|_2}{dx_i}=\frac{x_i}{\|x\|_2}$$

On the other hand, the derivative of the squared $l_2$-norm depends only on the element itself:

$$\frac{d\|x\|_2^2}{dx_i}=2x_i$$

This makes gradient based optimization much easier to implement.

$\endgroup$
1
$\begingroup$

An L2 regularizer corresponds to a Normal prior distribution $\mathcal{N}(\mathbf{0},\sigma\mathbf{I})$ with independent components, centered at zero. If you take the log of this prior, as you would when you calculate a Negative Log-Likelihood, you end up with $-\frac{1}{\sigma}||\mathbf{x}-\mathbf{0}||_2^2$

$\endgroup$
  • $\begingroup$ thanks, you mean prior for the residuals? $\endgroup$ – Haitao Du Aug 18 '16 at 15:25
  • $\begingroup$ I mean a prior distribution for the parameters that you are regularizing. $\endgroup$ – fstab Aug 18 '16 at 16:07
  • $\begingroup$ thanks for letting me know. I never know you can have prior for the parameters. Could you send me more related reference? $\endgroup$ – Haitao Du Aug 18 '16 at 16:19
  • 1
    $\begingroup$ stats.stackexchange.com/questions/163388/… $\endgroup$ – fstab Aug 19 '16 at 9:53
  • $\begingroup$ There's an extended discussion of the links between regularization and Bayesian statistics in Gelman's BDA3. $\endgroup$ – Sycorax Oct 12 '16 at 20:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.