What are the drawbacks of using least squares loss for regression? It seems to be like the most popular loss function for regression, for everything from OLS (it's in the name!) to sophisticated regularized regressions. 
Why is it so popular and what are the drawbacks?
 A: Ideally the loss function should reflect the losses that are caused to you by the forecast errors. So, in this ideal setup there are no advantages or disadvantages of loss functions so long they represent your losses appropriately. 
For instance, if any over or under prediction by $\Delta y$ units of items sold leads to $\$110\times (\Delta y)^2$ losses then there are no disadvantages of the $L(\Delta y)=(\Delta y)^2$ loss function. It's just the reality whether you like it or not. Using any other loss function would have been simply wrong, not advantageous or disadvantageous.
Unfortunately, almost nobody even tries to construct the true loss function these days. There could be many reasons why we don't do it anymore, but the practice is such that we choose loss functions based on convenience. This leads to them having advantages and disadvantages over each other.
So, for instance, if your error distribution is Cauchy, then the least squares loss function will lead you to nowhere, since they're linked to the expectations (moments) which do not exist for Cauchy. On the other hand, the least absolute values will produce a solution for Cauchy, since they're linked to the median which does exist for this distribution. In this regard the least squares are less robust than absolute values. On a related note, the least squares models are sensitive to outliers.
A: The second most popular choice to minimizing the squared distance (L2 loss) of predictions and targets is the absolute distance (L1 loss). 
The first big difference is that L2 loss places much more weight on outliers, because the squared distance is proportionally much bigger. 
The second big difference is the assumed distribution around the trend. L2 loss assumes the residuals are gaussian, and L1 loss assumes they are laplacian. See this discussion for more details.
