$L_p$ Norms - What is special about $p=2$?

Question

An $L_1$ norm is unique (at least partly) because $p=1$ is at the boundary between non-convex and convex. An $L_1$ norm is the 'most sparse' convex norm (right?).

I understand that the $p=2$ Euclidean norm has roots in geometry and it has a clear interpretation when dimensions have the same units. But I don't understand why it is used preferentially over other real numbers $p>1$: $p=1.5$? $p=\pi$? Why not use the full continuous range as a hyperparameter?

What am I missing?

Answer 1

A more mathematical explanation is that the space $l^p$, consisting of all series that converge in p-norm, is only Hilbert with $p=2$ and no other value. This means that this space is complete and the norm on that space may be induced by an inner product (think of the familiar dot-product in $R^n$), so it's a little nicer to work with.

Answer 2

Here are a couple of reasons:

1. It's related in a very special way to the inner product: it's its own dual norm (i.e. it's "self-dual").
   This means that, if you consider all vectors inside the $\ell_2$ unit ball, their maximum inner product with any vector $z$ is the $\ell_2$ norm of $z$ itself. Less fancily, it satisfies the property that $\lVert x\rVert_2^2 = x \cdot x$. No other $\ell_p$ norm behaves this way.

2. It has a very conveniently smooth gradient: $$\nabla_x\ \lVert f(x)\rVert_2^2 = 2 \ \nabla f(x) \cdot f(x)$$ You really can't beat that!

Answer 3

Though there can be many more reasons but AFAIK p=2 is preferred because of the following reasons :

• Measure of similarity/dissimilarity : For p=2, the Euclidean norm gives a measure of similarity or dissimilarity between two vectors which can then further be used for getting a better insight about the data. More detailed answers on this can be found here.
• Regularization : L2 norm is used for regularization in machine learning and is preferred because of two reasons- 1) It is easily differentiable 2) With L2 regularization, the weights tends to reduce in proportional to weights. Hence L2 regularization penalizes the bigger weights more as compared to the smaller weights.

Answer 4

Squared errors under linear models are often preferred because of:

• the relation to orthogonality, that behaves well with respect to some random phenomena considered as noise (uncorrelatedness)
• it is convex and differentiable, not $L_1$
• it yields tractable optimization algorithms as the derivative turns into linear systems

$L_1$ is often considered as a convenient proxy or convex relaxation to the strict sparsity (the count of non-zero terms) which is combinatorially complicated, see for instance For Most Large Underdetermined Systems of Linear Equations the Minimal $\ell_1$-norm Solution is also the Sparsest Solution. Some tend to use $\ell_p$, $0<p<1$ to enforce more sparsity, at the cost of "losing" convexity.

However, the $\ell_0$ count measure is insensitive to non-zero scaling. Multiply a vector by a non-zero constant, the number of non-zeros terms will remain the same. Thus, $\ell_0$ is $0$-order homogeneous, while $\ell_p$ norms or quasi-norms are all $1$-order homogeneous. Even if, somehow, $\ell_p \to \ell_0$ as $p\to 0$, this discrepancy seems a gap to me.

Thus, keeping with norms, some are considering (non-convex) norm ratios, such as $\ell_1 / \ell_2 $, see for instance the references in Euclid in a Taxicab: Sparse Blind Deconvolution with Smoothed $\ell_1 / \ell_2 $ Regularization.