15
$\begingroup$

MAD = Mean Absolute Deviation MSE = Mean Squared Error

I've seen suggestions from various places that MSE is used despite some undesirable qualities (e.g. http://www.stat.nus.edu.sg/~staxyc/T12.pdf, which states on p8 "It is commonly believed that MAD is a better criterion than MSE. However, mathematically MSE is more convenient than MAD.")

Is there more to it than that? Is there a paper that thoroughly analyzes the situations in which various methods of measuring forecast error are more/less appropriate? My google searches haven't revealed anything.

A similar question to this was asked at https://stackoverflow.com/questions/13391376/how-to-decide-the-forecasting-method-from-the-me-mad-mse-sde, and the user was asked to post on stats.stackexchange.com, but I don't think they ever did.

$\endgroup$
  • 2
    $\begingroup$ MAD is typically Median Absolute Deviation rather than Mean, no? $\endgroup$ – Brian D Nov 10 '17 at 17:54
  • $\begingroup$ @BrianD: in the wider statistics community, you are right. In the narrower forecasting community, "MAD" invariably is the "mean absolute deviation", AKA MAE. $\endgroup$ – Stephan Kolassa Sep 25 '18 at 9:26
22
$\begingroup$

To decide which point forecast error measure to use, we need to take a step back. Note that we don't know the future outcome perfectly, nor will we ever. So the future outcome follows a probability distribution. Some forecasting methods explicitly output such a full distribution, and some don't - but it is always there, if only implicitly.

Now, we want to have a good error measure for a point forecast. Such a point forecast $F_t$ is our attempt to summarize what we know about the future distribution (i.e., the predictive distribution) at time $t$ using a single number, a so-called functional of the future density. The error measure then is a way to assess the quality of this single number summary.

So you should choose an error measure that rewards "good" one number summaries of (unknown, possibly forecasted, but possibly only implicit) future densities.

The challenge is that different error measures are minimized by different functionals. The expected MSE is minimized by the expected value of the future distribution. The expected MAD is minimized by the median of the future distribution. Thus, if you calibrate your forecasts to minimize the MAE, your point forecast will be the future median, not the future expected value, and your forecasts will be biased if your future distribution is not symmetric.

This is most relevant for count data, which are typically skewed. In extreme cases (say, Poisson distributed sales with a mean below $\log 2\approx 0.69$), your MAE will be lowest for a flat zero forecast. See here or here or here for details.

I give some more information and an illustration in What are the shortcomings of the Mean Absolute Percentage Error (MAPE)? That thread considers the , but also other error measures, and it contains links to other related threads.


In the end, which error measure to use really depends on your Cost of Forecast Error, i.e., which kind of error is most painful. Without looking at the actual implications of forecast errors, any discussion about "better criteria" is basically meaningless.

Measures of forecast accuracy were a big topic in the forecasting community some years back, and they still pop up now and then. One very good article to look at is Hyndman & Koehler "Another look at measures of forecast accuracy" (2006).

Finally, one alternative is to calculate full predictive densities and assess these using proper .

$\endgroup$
  • $\begingroup$ Thanks for the response, and the link. I was not familiar with the term "Cost of Forecast Error". It seems like it relates to situations where (e.g.) a business is forecasting how many widgets it will sell, and perhaps the pain they suffer for overestimating is twice as much as the pain they suffer for underestimating. However, I'm mostly thinking about a context in which lay people are making forecasts about with no readily apparent cost of forecast error (e.g. "How many tweets will Bill Gates make in the next 5 months?"). In such a situation will my choice of error measure be arbitrary? $\endgroup$ – user1205901 Dec 13 '12 at 22:34
  • 3
    $\begingroup$ The Cost of Forecast Error has been discussed in the practitioner-oriented journal Foresight: forecasters.org/foresight Very much recommended! (Full disclosure: I'm an associate editor.) I agree that the CoFE is not readily apparent in your example, but then I would wonder how much effort you should really spend in optimizing your error measure... $\endgroup$ – Stephan Kolassa Dec 13 '12 at 22:42
4
$\begingroup$

The advantages of using MAE instead of MSE are explained in Davydenko and Fildes (2016), see Section 3.1:

...Some authors (e.g., Zellner, 1986) argue that the criterion by which we evaluate forecasts should correspond to the criterion by which we optimise forecasts. In other words, if we optimise estimates using some given loss function, we must use the same loss function for empirical evaluation in order to find out which model is better.

Fitting a statistical model usually delivers forecasts optimal under quadratic loss. This, e.g., happens when we fit a linear regression. If our density forecast from statistical modelling is symmetric, then forecasts optimal under quadratic loss are also optimal under linear loss. But, if we stabilise the variance by log-transformations and then transform back forecasts by exponentiation, we get forecasts optimal only under linear loss. If we use another loss, we must first obtain the density forecast using a statistical model, and then adjust our estimate given our specific loss function (see examples of doing this in Goodwin, 2000).

Let’s assume we want to empirically compare two methods and find out which method is better in terms of a symmetric linear loss (since this type of loss is commonly used in modelling). If we have only one time series, it seems natural to use a mean absolute error (MAE). Also, MAE is attractive as it is simple to understand and calculate (Hyndman, 2006)...

References

Davydenko, A., & Fildes, R. (2016). Forecast Error Measures: Critical Review and Practical Recommendations. In Business Forecasting: Practical Problems and Solutions. John Wiley & Sons

$\endgroup$
  • $\begingroup$ Could you give a full citation to the paper, rather than just "Davydenko and Fildes, 2016"? $\endgroup$ – Silverfish Feb 23 '16 at 12:13
  • $\begingroup$ We like our answers to be standalone, so that they are not adversely affected by links going dead. Do you think you could expand on your answer somewhat, to summarise what you thought were the key points of its content that are relevant to this question? Otherwise, this is really more suitable for a comment than an answer. (I appreciate you don't have enough reputation to post comments yet, but we can convert it into one for you.) $\endgroup$ – Silverfish Feb 23 '16 at 12:14
  • 1
    $\begingroup$ Thanks for your reply! Here's what (Davydenko and Fildes, 2016) says: Fitting a statistical model usually delivers forecasts optimal under quadratic loss. This, e.g., happens when we fit a linear regression. If our density forecast from statistical modelling is symmetric, then forecasts optimal under quadratic loss are also optimal under linear loss. But, if we stabilise the variance by log-transformations and then transform back forecasts by exponentiation, we get forecasts optimal only under linear loss. $\endgroup$ – Turbofly Feb 23 '16 at 12:21
  • 1
    $\begingroup$ Thanks! You can edit this information into your answer (the "edit" button is at the bottom of your post). $\endgroup$ – Silverfish Feb 23 '16 at 12:25
  • $\begingroup$ Thanks a lot. I've done some formatting and given a full citation. $\endgroup$ – Silverfish Feb 23 '16 at 18:12
3
$\begingroup$

Why not compare $RMSE = \sqrt{MSE}$ and $MAE = MAD$?

Actually,

$MAE \leq RMSE \leq \sqrt{n} MAE$ for regression models:

  • lower bound: each case contributes the same absolute amount of error $e$:
    $RMSE = \sqrt{\frac{1}{n} \sum e_i^2} = \sqrt{\frac{1}{n} n e^2} = e = MAE$
  • upper bound: a single case having error $e$ while all other cases have 0 error:
    $MAE = \frac{e}{n}$
    $RMSE = \sqrt{\frac{1}{n} \sum e_i^2} = \sqrt{\frac{1}{n} e^2} = \sqrt{\frac{1}{n} (n MAE)^2} = \sqrt{n} MAE$

($MAE \leq RMSE \leq \sqrt{MAE}$ for classification with partial class memberships $y_i$ and/or $\hat y_i$ are $\in [0, 1]$ -- i.e. they can actually take values in between 0 and 1).

  • upper bound: here, $e_i$ is $\leq 1$, so
    $MAE = \frac{n_{wrong}}{n}$
    $RMSE = \sqrt{\frac{1}{n} \sum e_i^2} = \sqrt{\frac{1}{n} n_{wrong}} = \sqrt{MAE}$
    (This upper bound occurs for integer $n_{wrong}$, if you go for partial/fractional class membership and thus also for $e_i \in [0, 1]$, things get a bit more complicated because you need to take into account that the maximum possible error can be less than 1, and you may have a "leftover" $e_i < 1$ which both lower the upper bound a bit further.)

If the RMSE is close the MAE, you have many small deviations, if it is close to its upper bound, there are few grossly wrong predictions.

$\endgroup$
  • $\begingroup$ do you mean sqrt(n)*MAE or sqrt(n*MAE) as an upper bound? $\endgroup$ – Chris Sep 30 '16 at 2:36
  • 1
    $\begingroup$ @Chris: it is sqrt (n) * MAE, see my edit. $\endgroup$ – cbeleites Sep 30 '16 at 14:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.