Is there any point in using MSE loss -- (a-b)^2 instead of L1 loss -- abs(a-b) in modern DNN/CNN architectures which use ReLU/ReLU-like activations? If so, why?
Suppose you want an unbiased prediction and that the conditional distribution of your dependent data is asymmetric. Then you want to minimize the squared error, or $L^2$ loss.
Minimizing the absolute error, or $L^1$ loss, is equivalent to finding the median of the conditional distribution (Hanley et al., 2001, The American Statistician), not the mean. If the distribution is asymmetric, this will typically mean that the output is biased.
This is a purely statistical effect. It is completely independent of your ML algorithm, NN architecture, fitting method etc.