Can I use mean absolute scaled error (MASE) from the accuracy function for time series cross validation?

Question

I am using the "forecast" package in R to forecast time series data. I am programming some time series cross validation based off of reading resources from Rob J Hyndman.

The last paragraph on page 7 in Hyndman "Measuring forecast accuracy" states:

"To calculate the MASE (sic, mean absolute scaled error) we need to compute the scaling statistic Q, but we do not want the value of Q to change with each training set. One approach is to compute Q using all the available data"

I interpret this as holding the mean absolute error (MAE) calculation for the naive forecast constant across all MASE Calculations. (Note: $\text{MASE}=\text{MAE}/Q$)

The accuracy function in the "forecast" package calculates MASE for a forecast and states (on page 4 of the manual):

By default, the MASE calculation is scaled using MAE of in-sample naive forecasts for nonseasonal time series, in-sample seasonal naive forecasts for seasonal time series and in-sample mean forecasts for non-time series data.

As I roll my forecast origin forward, can I run the accuracy function to calculate the MASE? Or do I have to manually calculate the MASE because the accuracy function does not use "all the available data" and will change for each training data set?

Answer 1

accuracy() uses the training sample on which a particular forecast is based. As you note, this will change in each iteration of your rolling origin evaluation. So indeed, you need to "roll your own".

This is less onerous than it looks like. Note that the MASE is the MAE or MAD, divided by some scaling factor. accuracy() will give you the MAE. (Of course you can use this in rolling origin evaluation.) So you only need to calculate a single scaling factor for each time series, which you will then apply to the rolling origin MAEs.

For instance, the "classical" scaling factor Hyndman & Koehler (2006) originally proposed is simply the in-sample MAE of the naive random walk forecast. If the data you want to base this on (e.g., the "common" history) is

foo <- ts(rnorm(1,20))

then you can calculate this very easily by

mean(abs(foo[-1]-foo[-20]))

---

(Note: it should be possible to use tail(foo,-1)-head(foo,-1), but this yields a vector of all zeros... I wonder whether something is buggy here, though both tail(foo,-1) and head(foo,-1) look fine. Weird.)

(Update: tail(foo,-1)-head(foo,-1) works on plain vanilla R, but not if I load the forecast 8.5 package. Sounds like a bug. I'll inform Rob Hyndman.)

(Update 2: I got the following answer back from Rob -

This is a feature, not a bug.

By defining head.ts and tail.ts, we are ensuring that head() and tail() on ts objects now retain their class (essential for plotting, etc.). By retaining the "ts" class, the - operation will now use Ops.ts. This will compute math operations with respect to time rather than with respect to position in the vector.

)