I have read through: http://robjhyndman.com/hyndsight/smape/ and https://www.otexts.org/fpp/2/5 and a lot of Hyndman, R. J. and Koehler, A. B. (2006) ‘Another look at measures of forecast accuracy’, International journal of forecasting, 22(4), pp. 679–688.

Having read those, I understand the reasons MASE exists. I also believe I understand how it works in the examples given, but the examples tend to be for similar problems in which the training data and test data are consecutive. In my case I have a year of training data, and I want to test a model based on the data using a different year which does not directly follow.

As I am not attempting to forecast one step ahead, it does not seem appropriate to me to use $Y_t = Y_{t-1}$ as the naive method.

  • Am I correct in thinking it is inappropriate?
  • If so, what would be appropriate?

  • One thought was that the denominator be the value recorded at that point in the training year: would that be more appropriate, or appropriate at all?

  • $\begingroup$ Your data aren't annual, right? In other words, you've got lots of observations within those years? If so, the fact that the periods aren't consecutive shouldn't matter. The forecasts from which MASE is computed will just pick up at the second time step in your test set, and everything works fine. $\endgroup$ – ulfelder Aug 8 '16 at 10:16
  • $\begingroup$ Yes, within the year I have data for each hour. What if the data were seasonal? $\endgroup$ – greenglass Aug 8 '16 at 10:53
  • $\begingroup$ To expand on that response, when I do use the usual method described MASE returns quite massive values, which is unsurprising given that the value at t is almost certainly going to be similar to the value at t-1 in most cases, but the value of t-1 isn't usually known (only predicted). So it seems a little unfair and wrong to use it here. $\endgroup$ – greenglass Aug 8 '16 at 16:25
  • $\begingroup$ I don't understand. In your test set, don't you know the value at t-1 for all but the first hourly observation? $\endgroup$ – ulfelder Aug 9 '16 at 9:46
  • $\begingroup$ In the test set, yes. But usually I will want to predict, e.g., 2017 using a model based on 2015. If I don't know t-1 for the actual data I want to predict, shouldn't I use something I will have when evaluating the model's performance against the test data? $\endgroup$ – greenglass Aug 9 '16 at 10:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.