What are good sign of fit from result of forecast::accuracy.

How to interpret

                  ME          RMSE      MAE       MPE     MAPE     MASE      ACF1

Training set -2.055155e-16 5.764161 4.322594 -8.302648 17.98444 6.244566 0.8651557

Test set      1.038893e+00 5.857035 4.353372 -4.400336 16.60394 6.289029        NA
  • $\begingroup$ Pass on this, but a bad sign is any report implying that we can and should be thinking about 7 significant figures. $\endgroup$ – Nick Cox Oct 29 '15 at 17:06
  • $\begingroup$ I guess most statistically minded people know two or three of these immediately from the abbreviations. Perhaps only forecasting experts will know all of them at first sight. I suppose that doesn't matter because it is the latter who should be able to produce good answers. But FWIW I flag the need to explain these to any but a highly trained readership. $\endgroup$ – Nick Cox Oct 29 '15 at 17:10
  • $\begingroup$ @NickCox: re your first comment, I agree that seven sig figs are too many, but believe me - the author of the forecast package knows what he's doing. $\endgroup$ – Stephan Kolassa Oct 29 '15 at 20:09
  • $\begingroup$ MASE is one of the most extremely confusing error metric, very difficult to explain to a non technical audience, use mean absolute error if you are comparing similar measure of units. MEan absolute percentage error (MAPE) or symmetric mean absolute percentage error (sMAPE) if you have different measure of units. $\endgroup$ – forecaster Oct 30 '15 at 2:04
  • $\begingroup$ I see no justification of using MASE, unless you have zeros in your forecast which in real world forecasting is almost non existent except intermittent demand forecasting. $\endgroup$ – forecaster Oct 30 '15 at 2:12

A MASE (Mean Absolute Scaled Error) of 6.24 in-sample is indeed a bit disconcerting. It means that your forecasting method yields in-sample absolute errors that are 6.24 times as large as those of a naive random walk model. This should not happen, unless you have a badly misspecified model.

This earlier thread on interpreting the MASE may be helpful.

In general, it is very hard to say whether a given error is "good enough" in forecasting. External benchmarks are pretty much useless, as there is just too much variation between series. I'd recommend that you simply try various approaches that model obvious structure in your data - if your series is obviously seasonal, a non-seasonal method won't be very helpful, and so on.

  • 1
    $\begingroup$ It does seem odd to see an MASE of 6. I note in the R forecast package documentation (of accuracy) that by default MASE uses a seasonal naive forecast for seasonal series. So it could be that the model fit is handling seasonality very incorrectly. $\endgroup$ – zbicyclist Oct 29 '15 at 20:22
  • $\begingroup$ MASE is unnecessary complication in this instance, why not MAE or sMAPE? Seems to be both are very apt for this type of problem. $\endgroup$ – forecaster Oct 30 '15 at 2:16
  • $\begingroup$ @forecaster: if you have a single series, then I agree that the MASE is less informative than the MAE, since the MASE is simply the MAE scaled by a factor that does not depend on the forecast (namely, the in-sample naive forecast MAE). The MASE makes sense once you have multiple series on different levels, where you can't very well compare "raw" MAEs. $\endgroup$ – Stephan Kolassa Oct 30 '15 at 7:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.