How to understand all the loss functions are the same? I remember I read it many times in CV and in other papers says that "All the loss functions are essentially the same", what is that mean and how to understand it intuitively?
 A: One sensible interpretation of this is that all "loss functions which are derived from finite dimensional norms" are the same. So if for pair $(x_i,y_i)$ your loss function has the form $L_i(x_i,y_i)=\|f(x_i)-y_i\|$, where $\|\cdot\|$ is a norm, then all loss functions become equivalent: there exist constants $c,C>0$ depending only on dimension, the pair of norms involved, and the number of data points, such that for two loss functions $L,L'$, one has:
$$cL'\leq L\leq CL'.$$
All such loss functions are within some maximal deformation of each-other and are topologically indistinguishable. In particular this implies that if your classifier can theoretically become perfect ($L=0$) under loss function $L$, then it will do so also under $L'$, perhaps at a different rate.
For example,
$$\|x\|_2\leq \|x\|_1\leq \sqrt{n}\|x\|_2.$$
However, the world we live in possesses dimensions, and if say $n=100$, then the two norms are within a factor of 10 of eachother, which from an operational point of view could be enormous.
