Why the default matrix norm is spectral norm and not Frobenius norm?

Question

For vector norm, the L2 norm or "Euclidean distance" is the widely used and intuitive definition. But why "most used" or "default" norm definition for a matrix is spectral norm, but not Frobenius norm (which is similar to L2 norm for vectors)?

Does that have something to do with iterative algorithms / matrix powers (if the spectral radius is smaller than 1, then the algorithm will converge)?

---

1. It is always arguable for the words like "most used" ,"default". The word "default" mentioned above is coming from the default return type in Matlab function norm. In R the default norm for matrix is L1 norm. Both of the are "unnatural" to me (for a matrix, it seems more "natural" to do $\sqrt{\sum_{i,j}a^{2}_{i,j}}$ like in vector). (Thanks for @usεr11852 and @whuber's comments and sorry for the confusion.)

2. May be expand the usage of the matrix norm would help me to understand more?

Answer 1

In general, I am unsure that the spectral norm is the most widely used. For example the Frobenius norm is used for to approximate solution on non-negative matrix factorisation or correlation/covariance matrix regularisation. I think that part of this question stems from the terminology misdemeanour some people do (myself included) when referring to the Frobenius norm as the Euclidean matrix norm. We should not because actually the $L_2$ matrix norm (ie. the spectral norm) is the one that is induced to matrices when using the $L_2$ vector norm. The Frobenius norm is that is element-wise: $||A||_F = \sqrt{\sum_{i,j}a_{i,j}^2}$, while the $L_2$ matrix norm ($||A||_2 = \sqrt{\lambda_{max}(A^T A)})$) is based on singular values so it is therefore more "universal" (for lack of a better term?). The $L_2$ matrix norm is a Euclidean-type norm since it is induced by the Euclidean vector norm, where $||A||_2 = \max\limits_{||x||_2 =1} || Ax||_2$. It therefore an induced norm for matrices because it is induced by a vector norm, the $L_2$ vector norm in this case.

Probably MATLAB aims to provide the $L_2$ norm by default when using the command norm; as a consequence it provides the Euclidean vector norm but also the $L_2$ matrix norm, ie. the spectral matrix norm (rather than the wrongly quoted "Frobenius/Euclidean matrix norm"). Finally let me note that what is the default norm is a matter of opinion to some extend: For example J.E. Gentle's "Matrix Algebra - Theory, Computations, and Applications in Statistics" literally has a chapter (3.9.2) named: "The Frobenius Norm - The “Usual” Norm"; so clearly the spectral norm is not the default norm for all parties considered! :) As commented by @amoeba, different communities might have different terminology conventions. It goes without saying that I think Gentle's book is an invaluable resource on the matter of Linear Algebra application in Statistics and I would prompt you to look it further!

Answer 2

A part of the answer may be related to numeric computing.

When you solve the system $$ Ax=b $$ in finite precision, you don't get the exact answer to that problem. You get an approximation $\tilde x$ due to the constraints of finite arithmetics, so that $A\tilde x \approx b$, in some suitable sense. What is it that your solution represents, then? Well, it may well be an exact solution to some other system like $$ \tilde A \tilde x = \tilde b $$ So for $\tilde x$ to have utility, the tilde-system must be close to the original system: $$ \tilde A \approx A, \quad \tilde b \approx b $$ If your algorithm of solving the original system satisfies that property, then it is referred to as backward stable. Now, the accurate analysis of how big the discrepancies $\tilde A-A$, $\tilde b-b$ are eventually leads to errors on bounds which are expressed as $\| \tilde A-A \|$, $\| \tilde b-b\|$. For some analyses, the $l_1$ norm (max column sum) is the easiest one to push through, for others, the $l_\infty$ norm (max row sum) is the easiest to push through (for components of the solution in the linear system case, for instance), and for yet others, the $l_2$ spectral norm is the most appropriate one (induced by the traditional $l_2$ vector norm, as pointed out in another answer). For the work horse of statistical computing in symmetric p.s.d. matrix inversion, Cholesky decomposition (trivia: the first sound is a [x] as in Greek letter "chi", not [tʃ] as in "chase"), the most convenient norm to keep track of the error bounds is the $l_2$ norm... although the Frobenius norm also pops up in some results e.g. on partitioned matrix inversion.

Answer 3

The answer to this depends on the field you're in. If you're a mathematician, then all norms in finite dimensions are equivalent: for any two norms $\|\cdot\|_a$ and $\|\cdot\|_b$, there exist constants $C_1,C_2$, which depend only on dimension (and a,b) such that:

$$C_1\|x\|_b\leq \|x\|_a\leq C_2\|x\|_b.$$

This implies that norms in finite dimensions are quite boring and there is essentially no difference between them except in how they scale. This usually means that you can choose the most convenient norm for the problem you're trying to solve. Usually you want to answer questions like "is this operator or procedure bounded" or "does this numerical process converge." With boundedness, you only usually care that something is finite. With convergence, by sacrificing the rate at which you have convergence, you can opt to use a more convenient norm.

For example, in numerical linear algebra, the Frobenius norm is sometimes preferred because it's a lot easier to calculate than the euclidean norm, and also that it naturally connects with a wider class of Hilbert Schmidt operators. Also, like the Euclidean norm, it's submultiplictive: $\|AB\|_F\leq \|A\|_F\|B\|_F$, unlike say, the max norm, so it allows you to easily talk about operator multiplication in whatever space you're working in. People tend to really like both the $p=2$ norm and the Frobenius norm because they have natural relations to both the eigenvalues and singular values of matrices, along with being submultiplictive.

For practical purposes, the differences between norms become more pronounced because we live in a world of dimensions and it usually matters how big a certain quantity is, and how it's measured. Those constants $C_1,C_2$ above are not exactly tight, so it becomes important just how much more or less a certain norm $\|x\|_a$ is compared to $\|x\|_b$.