4
$\begingroup$

I want to understand the importance of optimizing the correct loss function. Say that I am building a linear regression model $p$ for predicting some values $y_1,\ldots,y_n$.

I choose to fit my linear model such that it minimizes the mean squared errors. Now I send my model off to a statistical prediction competition, where they instead of using MSE as an error metric uses mean absolute error (so no squares). How will this affect my models predictive power? In general, what can be said about the importance of optimizing the "correct" loss function?

Edit: If one should put the question in a more specific context, then that context would be predictive modelling competitions like those featured on Kaggle.com. I want to understand the importance of choosing models and loss functions which correspond to the evaluation metric for the competition. One reason for this is this comment by the winner of a Kaggle competition.

$\endgroup$
  • 1
    $\begingroup$ The answers will depend on the distribution of the y's. Can you define what you mean by 'predictive power'? $\endgroup$ – Glen_b Jan 27 '14 at 13:23
  • $\begingroup$ I have updated the question in an attempt to specify more clearly why I am asking. $\endgroup$ – utdiscant Jan 27 '14 at 13:55
5
$\begingroup$

Say that I am building a linear regression model p for predicting some values $y_1,…,y_n$.

If the data contains a few extreme outliers in the response - or even just one - the MSE fitted equation can be pulled arbitrarily far away from the MAE one.

Consider the simplest regression model (just an intercept, $\alpha$), and following data:

  0.0003 0.0001 0.0002 0.0004 50000 0.0002 0.0004 0.0003 0.0001 0.0003

The MAD solution is $\alpha$ = 0.0003. The MSE solution is 5000.00023.

The MAD of the minimum MAD solution is about 0.0001. The MAD of the minimum MSE solutions is about 5000. You can potentially do very badly, if you use MSE when the criterion is MAD.

$\endgroup$
  • $\begingroup$ I think this goes too much into the direction of robust methods; if we can expect outliers of this magnitude in the next data set the MSE predictions may still be better when it comes to minimizing the MSE of the predictions. $\endgroup$ – Erik Jan 27 '14 at 14:30
  • 1
    $\begingroup$ In respect of the question being asked, "what can be said about the importance of optimizing the "correct" loss function?" my example addresses that question very clearly and directly by showing that the two can be arbitrarily different. $\endgroup$ – Glen_b Jan 27 '14 at 14:33
  • 1
    $\begingroup$ This suggests that it may make a huge difference. In the absence of clearer indication of what situations we might face, considering worst cases is a reasonable response. $\endgroup$ – Glen_b Jan 27 '14 at 14:46
  • $\begingroup$ Good point, though I read "correct" more as being the one that one is interested later on than the one that will yield a robust model for the particular usecase. Otherwise penalization is an excellent example for optimizing a different error metric than the one which one uses to measure the quality of the predictions on new data, which I felt was not was the OP was after by my reading from the original post. $\endgroup$ – Erik Jan 27 '14 at 14:53
  • $\begingroup$ @Erik I didn't choose to discuss the comparison of MSE with mean absolute error 'because it is more robust', I used that comparison because it's the specific pair of criteria mentioned in the second paragraph of the question. The OP posits the case where he uses MSE and the criterion of the competition is MAE and asks about the importance of using the correct one (MAE). I show that it can potentially be very important to use MAE, because MSE can be very bad when you are being judged by how you do on MAE. $\endgroup$ – Glen_b Jan 27 '14 at 15:20
1
$\begingroup$

I think this question is hard to answer in general and depends heavily on the data that is analyzed.

Simple example: have your model with only the intercept, e.g. we have no predictors and can just estimate the mean and predict that for all new data. If $y$ is normally distributed the sample mean (i.e. the MSE estimate) is best, even if you test new data with the MAE. That's because to minimize MAE you would always predict the median, and with the normal distribution we have median = mean and the MSE estimate of the mean has lower variance than the median - this means the sample mean is a better estimator for the median than the sample median.

This means we have constructed an situation in which optimizing the "wrong" loss function has helped us. OTH it is also easy to construct a situation where it will give suboptimal results. For example, take a probability distribution with point weights of $\frac{999}{1000}$ on 0 and $\frac{1}{1000}$ on 1. The sample mean will be around $\frac{1}{1000}$ for many sample. Assume we correctly estimate the mean to be 0.001 When we predict this the MAE will be $$ \frac{999}{1000}\frac{1}{1000}+\frac{1}{1000}\frac{999}{1000}=\frac{1998}{1000000} $$

However, we will almost certainly correctly estimate the median to be 0, the MAE is only $$ \frac{999}{1000}0+\frac{1}{1000}1=\frac{1}{1000} $$ That's about half the MAE compared to predicting the mean. So the general answer would be it depends.

$\endgroup$
0
$\begingroup$

Often closed form - solution is reason for preferring certain kinds of loss functions. Or it was since now we have computers for solving numerical problems.

If you have loss function with mean absolute errors, that leads to much more difficult problem than error squared as in least squares method.

And problem is also that it might not be possible to test between non-nested loss functions. That is reason why we have so many estimation methods for the same kind of problem! :)

$\endgroup$
  • 3
    $\begingroup$ No downvote, but I don't feel this adresses the question. $\endgroup$ – Erik Jan 27 '14 at 11:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.