Out of sample MASE

Question

When calculating the MASE, the original paper suggests using the in-sample naive forecast error for scaling of the out of sample forecast error.

When i use the the MAE generated by a naive forecast on the out of sample dataset however, I get a MASE which correlates more with the actual performance of the forecast in the tested period.

My understanding is that one limitation with using the out of sample naive MAE is that if the out of sample set is small, it is not reliable. This is however not the case in my application.

intuitively it seems more relevant to scale the predictions errors with the naive forecast errors of the same timeperiod.

My question:

Is it a good idea to use the MAE of the out of sample data to scale the forecast error? ie.

$$MASE=\frac{MAE}{MAE_{out-sample, \, naive}}$$

Answer 1

You can use whatever benchmark you are most comfortable with in the denominator. (A frequent choice is a naive seasonal forecast in-sample.) Your approach has the advantage of comparing your forecasts to the benchmark out-of-sample, which is probably more relevant than comparing it to the benchmark in-sample, as the "normal" MASE does.

It would just be a good idea to note explicitly what denominator you are using, so other people (or you yourself in half a year) don't get confused that their calculated "plain vanilla" MASE does not match your calculation.