# Is there any point in using MSE loss in modern deep neural networks?

Is there any point in using MSE loss -- (a-b)^2 instead of L1 loss -- abs(a-b) in modern DNN/CNN architectures which use ReLU/ReLU-like activations? If so, why?

• is there any reason to use L2 over L1 loss for a linear regression problem? Jun 7 '18 at 22:16
• To add some flavor to shiamo's comment, what context does the use of leaning algorithm (DNNs vs boosting vs regression) add to this kind of decision? What leads you to believe that DNNs would alter the appropriate choice of loss function? Jun 8 '18 at 2:33

Suppose you want an unbiased prediction and that the conditional distribution of your dependent data is asymmetric. Then you want to minimize the squared error, or $L^2$ loss.
Minimizing the absolute error, or $L^1$ loss, is equivalent to finding the median of the conditional distribution (Hanley et al., 2001, The American Statistician), not the mean. If the distribution is asymmetric, this will typically mean that the output is biased.
• @CagdasOzgenc: regarding your first point, I explicitly started with "Suppose you want an unbiased prediction". Of course if you have a different loss function than $L^2$, then you shouldn't use $L^2$. I see people use $L^1$ and then complain that their forecast is biased. Regarding your second point, if you know your second system, then you may be able to correct for such effects, e.g., adding a $e^\frac{\sigma^2}{2}$ factor in lognormal models. Plus, in such a situation you should tailor your loss to the system, anyway - but the OP seems to suggest that $L^1$ is good in general. Jun 8 '18 at 7:56
• I don't fully understand what you mean by "quadratic" and "arithmetic" mean. I assume you refer to $L^2$ and $L^1$ loss. Here is the problem: your realizations, conditional on your predictors, are random. They have a conditional distribution. Loss functions come in if you try to summarize this conditional distribution with a single number fit (or forecast). (It's better to model the entire density and use scoring-rules). $L^1$ loss is minimized by the median of the density, $L^2$ by the mean. ... Jun 10 '18 at 19:33
• ... If your conditional density is asymmetric, this means that the median is different from the mean. If you want the conditional median, great, use $L^1$ loss. However, usually people want an unbiased fit or forecast, and then the conditional median may not be correct. Here is an example: intermittent time series have "many" zeros, e.g., more than half the realizations. That is, the median is zero. Lots of people did not understand why a flat zero forecast was "optimal" under $L^1$ loss (Kolassa, 2016, IJF). Jun 10 '18 at 19:35