0
$\begingroup$

I am using the mean absolute error

mean(abs(obs - pred))

as one of the measures assessing the fit of my model. I would also like to have a standardised measure ranging 0 - 1 to compliment this. Given that there is MSE and SMSE, how does one go about to get a standardised MAE?

thank you.

$\endgroup$

3 Answers 3

2
$\begingroup$

If you can deduce the worst possible MAE in your particular situation, you can divide the MAE you actually get by this, which will scale your MAE to the interval [0,1], with a perfect fit mapped to 0 and the worst possible MAE mapped to 1.

However, often there is no upper bound to the MAE. Fits can often in principle be unboundedly bad. In such a situation, you cannot scale your MAE to any predetermined interval linearly. Of course you could non-linearly scale it by

$$ \text{MAE} \mapsto \frac{2}{\pi}\arctan(\text{MAE}), $$

which does map any MAE to [0,1], but I would rather doubt that this would be very enlightening.

$\endgroup$
1
$\begingroup$

The usual way of standardizing mean squared error is dividing by the variance of target variable mean((obs - pred)^2)/mean(obs^2), while for mean absolute error, you usually divide by the mean absolute deviation mean(abs(obs - pred))/mean(abs(obs)). This however does not give you guarantees that the result will be mapped to unit interval, just shows the errors on scale that is relative to the deviations of the target variable.

$\endgroup$
3
  • 1
    $\begingroup$ Why is mean(abs(obs)) the mean absolute deviation? And you provided a source how SMSE is defined but more interesting would be a source defining SMAE which was the question. $\endgroup$
    – user213325
    Commented Apr 10, 2019 at 11:21
  • 1
    $\begingroup$ @igoR87 Because you want to compare how the predictions deviate vs how much the data deviates, so you should have similar metrics as in numerator and denominator. $\endgroup$
    – Tim
    Commented Apr 10, 2019 at 11:27
  • 1
    $\begingroup$ That sounds reasonable. I wonder why I couldn't find any source with this defintion but with other definitions as posted in my answer. Although as you pointed out the definitions I found are problematic. It is strange since the measure SMAE is mentioned sometimes but there should be somewhere source for the definition $\endgroup$
    – user213325
    Commented Apr 10, 2019 at 11:39
0
$\begingroup$

I couldn't find any sources defining the SMAE when I looked for standardised mean absolute error but I did find some when I typed standardized mean absolute error. This paper for example defines SMAE as

mean(abs(obs-pred)) / mean(obs)

But the range wouldn't be 0-1 since you get negative values in cases where MAE is not zero and mean(obs) is negative. Further, there are other definitions of SMAE which for example suggest to devide MAE by the sd rather than by the mean.

EDIT

As pointed out by Tim the definition of the first source is problematic because mean(obs) could be zero which would mean deviding by zero. I add it to the answer in case someone doesn't see the comment right away.

$\endgroup$
7
  • $\begingroup$ What is mean(obs) is zero? Moreover, this does not have to be in [0, 1] as you don't have any guarantees that that the absolute error is smaller then mean. $\endgroup$
    – Tim
    Commented Apr 10, 2019 at 10:47
  • $\begingroup$ I didn't make thing up but posted definitions with the sources. Anyway, I didn't claim this would be the perfect answer. The opposite is the case since I mentioned there are contradicting definitions and I also already mentioned that range does not have to be [0, 1] in definition(s) provided. $\endgroup$
    – user213325
    Commented Apr 10, 2019 at 11:16
  • $\begingroup$ The metric you quote is in spirit of coefficient of variation: en.wikipedia.org/wiki/Coefficient_of_variation or stats.stackexchange.com/questions/118497/… $\endgroup$
    – Tim
    Commented Apr 10, 2019 at 11:44
  • $\begingroup$ This is kind of true although in case of CV one uses sd of one variable instead of mean(abs(obs-pred)). Anyway, your point that mean(obs) being zero is problematic and something I dind't have in mind while quoting the source. But I added it to my answer. Still, there must be some book or another source defining correctly the MSAE which I couldn't find what surprises me. $\endgroup$
    – user213325
    Commented Apr 10, 2019 at 11:55
  • $\begingroup$ @Tim: Should I delete my answer since it seems to be wrong? Sorry to use comments like a chat but I'm unsure because I'm relatively new to stackexchange. $\endgroup$
    – user213325
    Commented Apr 10, 2019 at 12:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.