13
$\begingroup$

An $L_1$ norm is unique (at least partly) because $p=1$ is at the boundary between non-convex and convex. An $L_1$ norm is the 'most sparse' convex norm (right?).

I understand that the $p=2$ Euclidean norm has roots in geometry and it has a clear interpretation when dimensions have the same units. But I don't understand why it is used preferentially over other real numbers $p>1$: $p=1.5$? $p=\pi$? Why not use the full continuous range as a hyperparameter?

What am I missing?

$\endgroup$
  • 1
    $\begingroup$ "Used preferentially" in which applications, specifically? Norms are ubiquitous in mathematics, statistics, and physics; in some subfields some norms are more prevalent than others because they are more meaningful or simpler to work with. For this reason, answers to this question will likely be numerous and varied (so varied, indeed, that personally I find this unanswerable). I have therefore made this a "Community Wiki" (CW) post; but if you have a specific application or narrow field in mind, then by making your question more precise it should be possible to remove the CW status. $\endgroup$ – whuber Mar 17 '17 at 12:41
12
$\begingroup$

A more mathematical explanation is that the space $l^p$, consisting of all series that converge in p-norm, is only Hilbert with $p=2$ and no other value. This means that this space is complete and the norm on that space may be induced by an inner product (think of the familiar dot-product in $R^n$), so it's a little nicer to work with.

$\endgroup$
4
$\begingroup$

Here are a couple of reasons:

  1. It's related in a very special way to the inner product: it's its own dual norm (i.e. it's "self-dual").
    This means that, if you consider all vectors inside the $\ell_2$ unit ball, their maximum inner product with any vector $z$ is the $\ell_2$ norm of $z$ itself. Less fancily, it satisfies the property that $\lVert x\rVert_2^2 = x \cdot x$. No other $\ell_p$ norm behaves this way.

  2. It has a very conveniently smooth gradient: $$\nabla_x\ \lVert f(x)\rVert_2^2 = 2 \ \nabla f(x) \cdot f(x)$$ You really can't beat that!

$\endgroup$
2
$\begingroup$

Though there can be many more reasons but AFAIK p=2 is preferred because of the following reasons :

  • Measure of similarity/dissimilarity : For p=2, the Euclidean norm gives a measure of similarity or dissimilarity between two vectors which can then further be used for getting a better insight about the data. More detailed answers on this can be found here.
  • Regularization : L2 norm is used for regularization in machine learning and is preferred because of two reasons- 1) It is easily differentiable 2) With L2 regularization, the weights tends to reduce in proportional to weights. Hence L2 regularization penalizes the bigger weights more as compared to the smaller weights.
$\endgroup$
1
$\begingroup$

Squared errors under linear models are often preferred because of:

  • the relation to orthogonality, that behaves well with respect to some random phenomena considered as noise (uncorrelatedness)
  • it is convex and differentiable, not $L_1$
  • it yields tractable optimization algorithms as the derivative turns into linear systems

$L_1$ is often considered as a convenient proxy or convex relaxation to the strict sparsity (the count of non-zero terms) which is combinatorially complicated, see for instance For Most Large Underdetermined Systems of Linear Equations the Minimal $\ell_1$-norm Solution is also the Sparsest Solution. Some tend to use $\ell_p$, $0<p<1$ to enforce more sparsity, at the cost of "losing" convexity.

However, the $\ell_0$ count measure is insensitive to non-zero scaling. Multiply a vector by a non-zero constant, the number of non-zeros terms will remain the same. Thus, $\ell_0$ is $0$-order homogeneous, while $\ell_p$ norms or quasi-norms are all $1$-order homogeneous. Even if, somehow, $\ell_p \to \ell_0$ as $p\to 0$, this discrepancy seems a gap to me.

Thus, keeping with norms, some are considering (non-convex) norm ratios, such as $\ell_1 / \ell_2 $, see for instance the references in Euclid in a Taxicab: Sparse Blind Deconvolution with Smoothed $\ell_1 / \ell_2 $ Regularization.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.