5
$\begingroup$

I applied two regression models (ordinary least square (OLS) and linear absolute regression) to the same dataset, where this dataset is split into train and test sets.

Two performance measures are used to check the accuracy of linear regression models:

MSE stands for mean square error.

MAD stands for mean absolute deviation.

I found that the model fit by OLS will have a lower MSE value on unseen data than one fit using linear absolute regression? 

On the other hand, the model that is fitted by linear absolute regression will have a lower MAD value on unseen data than the one that is fitted using OLS.

Therefore, if I use the MSE as a regression performance measure, I will end up saying that the OLS model is the best, and contradictorily, if I use MAD as a regression performance measure, I will say that absolute linear regression is the best?

There is a claim by my colleagues that performance metrics will prefer regression models with the same type of loss minimization. In other words, if I have a comparison study, I can't use MAD alone and say that this absolute linear regression is the best choice because if I report MSE, the OLS model is better.

In my recent question, Dave's answer showed counter-examples on both real and simulated datasets.

My question is as follows:

If their claim is not correct how would you phrase a counter-argument in a few sentences and in a logical manner, not only R codes? If they are correct, why does this happen and how?

$\endgroup$
6
  • 4
    $\begingroup$ Since “bias” has a technical meaning in statistics, it might be helpful to explain that with a different word (e.g., “preference”). $\endgroup$
    – Dave
    Commented Apr 2, 2022 at 18:08
  • 4
    $\begingroup$ That's not bias. It's using a scoring function that is consistent for a given functional. If you're not willing to predict whole distributions, then there is no theoretical way to derive a "universal" measure: your choice of a scoring function is a substantive decision that expresses an implicit interest in predicting a specific functional of the distribution, and using inconsistent scoring functions simply leads to improper scoring. $\endgroup$
    – Chris Haug
    Commented Apr 2, 2022 at 19:09
  • $\begingroup$ Related: stats.stackexchange.com/questions/470626 $\endgroup$ Commented Apr 3, 2022 at 7:25
  • 1
    $\begingroup$ Feels like nice question, could use better title. $\endgroup$
    – Nuclear241
    Commented Apr 3, 2022 at 7:50
  • 1
    $\begingroup$ Is there something lacking in the existing answers? I’d like to address any remaining concerns you have and close out this question. $\endgroup$
    – Dave
    Commented Apr 8, 2022 at 10:39

2 Answers 2

6
$\begingroup$

Dave's answer has nothing to do with whether there's bias in an in-sample metric vs. an out-of-sample metric when the algorithm optimizes the in-sample metric. His answer addresses whether minimizing the in-sample metric necessarily also minimizes the (expected) out-of-sample metric (Edit: it doesn't); it says nothing about the relative values of the two. The bias issue states that if you do minimize an in-sample metric, the corresponding out-of-sample metric can be expected to be worse; it says nothing about whether some other objective function could improve the OOS metric.

$\endgroup$
8
  • 2
    $\begingroup$ I’d argue that the whole point of my answer is that training with a different loss function than we use out-of-sample might result in better out-of-sample performance. $\endgroup$
    – Dave
    Commented Apr 2, 2022 at 18:01
  • 4
    $\begingroup$ @Dave - agreed, but it has nothing to do with the bias issue. Added an edit to include your point. $\endgroup$
    – jbowman
    Commented Apr 2, 2022 at 18:03
  • 2
    $\begingroup$ @jbowman, the idea of my main question is more simple. Is it correct that on the same dataset, if I use the MSE as a regression performance measure, I will end up saying that the OLS model is the best, and contradictorily, if I use MAD as a regression performance measure, I will say that absolute linear regression is the best? If yes or no, why does this happen and how? $\endgroup$
    – jeza
    Commented Apr 3, 2022 at 2:20
  • 2
    $\begingroup$ @jeza you can create any dataset for which the metric for out-of-sample data prefers any method of your choice, because it is out-of-sample, and doesn't have to follow the same distribution as the in-sample dataset. Perhaps the effect you're looking for is with the assumption that the out of sample follows the same distribution really closely as the in-sample. But then it's obvious that if the metric is the same as the optimized metric, it will be the best, since it's essentially the same data. $\endgroup$
    – justhalf
    Commented Apr 3, 2022 at 10:35
  • 2
    $\begingroup$ @justhalf, you are not getting the essence of it. Optimal training loss function does not generally coincide with the evaluation loss function, even under ideal conditions where the training sample comes from exactly the same data generating process as the evaluation sample. This is a fascinating fact; it is worth trying to understand it. A good starting point is estimating the true median (which is the minimizer of the expected absolute loss in the evaluation sample) by the sample mean (which is the minimizer of square loss in the training sample) – NOT the median – when the DGP is normal. $\endgroup$ Commented Apr 6, 2022 at 7:45
0
$\begingroup$

Since the out-of-sample data are different from the in-sample data, all bets are off when it comes to what the out-of-sample metric chooses as its preferred model. In some sense, we are tuning the in-sample loss function as a hyperparameter in order to achieve the best out-of-sample performance on our metric of choice. If the out-of-sample metric prefers a model trained with a different loss function, so be it! That’s why we tune the hyperparameter.

I would present the argument like that. I also would be comfortable giving simulations or empirical data where, for example, out-of-sample square loss prefers a model that was trained with absolute loss than with square loss (such as the examples I gave with Iris and simulations).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.