Is there any point in using MSE loss -- (a-b)^2 instead of L1 loss -- abs(a-b) in modern DNN/CNN architectures which use ReLU/ReLU-like activations? If so, why?

  • 1
    $\begingroup$ is there any reason to use L2 over L1 loss for a linear regression problem? $\endgroup$
    – shimao
    Jun 7 '18 at 22:16
  • 2
    $\begingroup$ To add some flavor to shiamo's comment, what context does the use of leaning algorithm (DNNs vs boosting vs regression) add to this kind of decision? What leads you to believe that DNNs would alter the appropriate choice of loss function? $\endgroup$ Jun 8 '18 at 2:33

Suppose you want an unbiased prediction and that the conditional distribution of your dependent data is asymmetric. Then you want to minimize the squared error, or $L^2$ loss.

Minimizing the absolute error, or $L^1$ loss, is equivalent to finding the median of the conditional distribution (Hanley et al., 2001, The American Statistician), not the mean. If the distribution is asymmetric, this will typically mean that the output is biased.

This is a purely statistical effect. It is completely independent of your ML algorithm, NN architecture, fitting method etc.

  • $\begingroup$ So what if the output is not converging to the mean? Unless your domain loss requires the mean as the optimal point, it is not very relevant. Besides if we are going to feed this output as an input to another system unless the output is gaussian and the system that it is fed into is linear then you will end up with a bias anyways at the final output. $\endgroup$ Jun 8 '18 at 6:17
  • $\begingroup$ @CagdasOzgenc: regarding your first point, I explicitly started with "Suppose you want an unbiased prediction". Of course if you have a different loss function than $L^2$, then you shouldn't use $L^2$. I see people use $L^1$ and then complain that their forecast is biased. Regarding your second point, if you know your second system, then you may be able to correct for such effects, e.g., adding a $e^\frac{\sigma^2}{2}$ factor in lognormal models. Plus, in such a situation you should tailor your loss to the system, anyway - but the OP seems to suggest that $L^1$ is good in general. $\endgroup$ Jun 8 '18 at 7:56
  • $\begingroup$ Sorry, can you please explain in simpler terms why looking for quadratic mean is supposed to be superior to looking for arithmetic mean? If what you want as an end result is to minimize expected L1 difference between truth and expected predictions based on test data. I didn't get it. Also, I'd like to note that L1 loss seems to me a lot more stable and less explosive, compared to the quadratic L2 loss, which seems to me like a really good reason for preference. As we know, with large loss values there's a large chance of exploding the network and getting "NaN" weights. $\endgroup$
    – Íhor Mé
    Jun 10 '18 at 18:27
  • $\begingroup$ I don't fully understand what you mean by "quadratic" and "arithmetic" mean. I assume you refer to $L^2$ and $L^1$ loss. Here is the problem: your realizations, conditional on your predictors, are random. They have a conditional distribution. Loss functions come in if you try to summarize this conditional distribution with a single number fit (or forecast). (It's better to model the entire density and use scoring-rules). $L^1$ loss is minimized by the median of the density, $L^2$ by the mean. ... $\endgroup$ Jun 10 '18 at 19:33
  • $\begingroup$ ... If your conditional density is asymmetric, this means that the median is different from the mean. If you want the conditional median, great, use $L^1$ loss. However, usually people want an unbiased fit or forecast, and then the conditional median may not be correct. Here is an example: intermittent time series have "many" zeros, e.g., more than half the realizations. That is, the median is zero. Lots of people did not understand why a flat zero forecast was "optimal" under $L^1$ loss (Kolassa, 2016, IJF). $\endgroup$ Jun 10 '18 at 19:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.