Importance of optimizing the correct loss function

Question

I want to understand the importance of optimizing the correct loss function. Say that I am building a linear regression model $p$ for predicting some values $y_1,\ldots,y_n$.

I choose to fit my linear model such that it minimizes the mean squared errors. Now I send my model off to a statistical prediction competition, where they instead of using MSE as an error metric uses mean absolute error (so no squares). How will this affect my models predictive power? In general, what can be said about the importance of optimizing the "correct" loss function?

Edit: If one should put the question in a more specific context, then that context would be predictive modelling competitions like those featured on Kaggle.com. I want to understand the importance of choosing models and loss functions which correspond to the evaluation metric for the competition. One reason for this is this comment by the winner of a Kaggle competition.

Answer 1

Say that I am building a linear regression model p for predicting some values $y_1,…,y_n$.

If the data contains a few extreme outliers in the response - or even just one - the MSE fitted equation can be pulled arbitrarily far away from the MAE one.

Consider the simplest regression model (just an intercept, $\alpha$), and following data:

  0.0003 0.0001 0.0002 0.0004 50000 0.0002 0.0004 0.0003 0.0001 0.0003

The MAD solution is $\alpha$ = 0.0003. The MSE solution is 5000.00023.

The MAD of the minimum MAD solution is about 0.0001. The MAD of the minimum MSE solutions is about 5000. You can potentially do very badly, if you use MSE when the criterion is MAD.

Answer 2

I think this question is hard to answer in general and depends heavily on the data that is analyzed.

Simple example: have your model with only the intercept, e.g. we have no predictors and can just estimate the mean and predict that for all new data. If $y$ is normally distributed the sample mean (i.e. the MSE estimate) is best, even if you test new data with the MAE. That's because to minimize MAE you would always predict the median, and with the normal distribution we have median = mean and the MSE estimate of the mean has lower variance than the median - this means the sample mean is a better estimator for the median than the sample median.

This means we have constructed an situation in which optimizing the "wrong" loss function has helped us. OTH it is also easy to construct a situation where it will give suboptimal results. For example, take a probability distribution with point weights of $\frac{999}{1000}$ on 0 and $\frac{1}{1000}$ on 1. The sample mean will be around $\frac{1}{1000}$ for many sample. Assume we correctly estimate the mean to be 0.001 When we predict this the MAE will be $$ \frac{999}{1000}\frac{1}{1000}+\frac{1}{1000}\frac{999}{1000}=\frac{1998}{1000000} $$

However, we will almost certainly correctly estimate the median to be 0, the MAE is only $$ \frac{999}{1000}0+\frac{1}{1000}1=\frac{1}{1000} $$ That's about half the MAE compared to predicting the mean. So the general answer would be it depends.

Answer 3

Often closed form - solution is reason for preferring certain kinds of loss functions. Or it was since now we have computers for solving numerical problems.

If you have loss function with mean absolute errors, that leads to much more difficult problem than error squared as in least squares method.

And problem is also that it might not be possible to test between non-nested loss functions. That is reason why we have so many estimation methods for the same kind of problem! :)