2
$\begingroup$

I use a memory-based learning model to predict human scores in a [0, 10] range (quiz results). As a forecast error metric I use Mean Absolute Error. I was wondering what is the relation between MAE and data distribution. How can i use the data (or the error) distribution to explain the error and possibly argue whether it is acceptable. For example, can a MAE of 60% be characterized as "good" for a non-normal data distribution rather than a normal one? Since i am new to statistics, I would dearly appreciate any suggested reference to related bibliography. Thank you in advance.

$\endgroup$
2
$\begingroup$

I don't think there is anything published on this. You could always work with simulations: simulate normally distributed data with a given mean $\mu$ and variance $\sigma^2$ and assess the MAE of the perfect forecast $\mu$ (which you know in this case).

Yes, for the normal distribution, you don't really need to simulate - you can easily calculate the expected MAE. But simulations are more easily adopted to other distributions.

So now you can give the minimum possible expected MAE for a time series with a given residual variance $\sigma^2$. You would still need to argue that for a given time series, you actually know the residual variance - that is, you need to argue that you cannot explain any more variance. This is hard, because with more information about the data generating process, you may indeed be able to explain more variance - it may just become unrealistic.

Note incidentally that minimizing the MAE may lead to biased forecasts for non-normal (specifically: asymmetric) distributions.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.