0
$\begingroup$

What are good sign of fit from result of forecast::accuracy.

How to interpret

                  ME          RMSE      MAE       MPE     MAPE     MASE      ACF1

Training set -2.055155e-16 5.764161 4.322594 -8.302648 17.98444 6.244566 0.8651557

Test set      1.038893e+00 5.857035 4.353372 -4.400336 16.60394 6.289029        NA
Improve this question
$\endgroup$
11
  • 1
    $\begingroup$ Pass on this, but a bad sign is any report implying that we can and should be thinking about 7 significant figures. $\endgroup$
    – Nick Cox
    Commented Oct 29, 2015 at 17:06
  • 1
    $\begingroup$ @BigBen: that is an extremely good question. First, I'll assume that zeros are not erroneous or abnormal (in which case you should not be calculating accuracy off them in the first place). So consider intermittent demand, which is count data with "many" zeros. Here is the problem: the MASE is a scalar multiple of the absolute error, and you minimize the AE by forecasting the conditional median. If you have more than half zeros, then your MASE-optimal forecast is a flat zero. ... $\endgroup$
    – Stephan Kolassa
    Commented Oct 24, 2023 at 15:29
  • 1
    $\begingroup$ @BigBen: ... This is presumably not what you want, which is why you probably should not be using the MASE in the first place. Turn this around: you should first figure out which functional of the unknown future distribution you want to elicit, and only then decide on your error measure. Want the conditional mean? Use the MSE (or a variation thereof). Want the conditional median? Use the MAE (or the MASE). And so forth. The argument in this thread is absolutely similar. ... $\endgroup$
    – Stephan Kolassa
    Commented Oct 24, 2023 at 15:31
  • 1
    $\begingroup$ @BigBen: ... See also this paper and this paper for more on this argument. Feel free to contact me on LinkedIn or ResearchGate for the papers if you are interested. $\endgroup$
    – Stephan Kolassa
    Commented Oct 24, 2023 at 15:34
  • 1
    $\begingroup$ @StephanKolassa got it! I had already read the first paper (100% agree) and thanks for the second. Thank you for your thorough work in this regard. $\endgroup$
    – BigBen
    Commented Oct 24, 2023 at 15:48

1 Answer 1

Reset to default
5
$\begingroup$

A MASE (Mean Absolute Scaled Error) of 6.24 in-sample is indeed a bit disconcerting. It means that your forecasting method yields in-sample absolute errors that are 6.24 times as large as those of a naive random walk model. This should not happen, unless you have a badly misspecified model.

This earlier thread on interpreting the MASE may be helpful.

In general, it is very hard to say whether a given error is "good enough" in forecasting. External benchmarks are pretty much useless, as there is just too much variation between series. I'd recommend that you simply try various approaches that model obvious structure in your data - if your series is obviously seasonal, a non-seasonal method won't be very helpful, and so on.

Improve this answer
$\endgroup$
3
  • 1
    $\begingroup$ It does seem odd to see an MASE of 6. I note in the R forecast package documentation (of accuracy) that by default MASE uses a seasonal naive forecast for seasonal series. So it could be that the model fit is handling seasonality very incorrectly. $\endgroup$
    – zbicyclist
    Commented Oct 29, 2015 at 20:22
  • $\begingroup$ MASE is unnecessary complication in this instance, why not MAE or sMAPE? Seems to be both are very apt for this type of problem. $\endgroup$
    – forecaster
    Commented Oct 30, 2015 at 2:16
  • $\begingroup$ @forecaster: if you have a single series, then I agree that the MASE is less informative than the MAE, since the MASE is simply the MAE scaled by a factor that does not depend on the forecast (namely, the in-sample naive forecast MAE). The MASE makes sense once you have multiple series on different levels, where you can't very well compare "raw" MAEs. $\endgroup$
    – Stephan Kolassa
    Commented Oct 30, 2015 at 7:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.