MAPE results for the 4-week post-sample period

Question

I'm trying to get the same results reported in the paper Taylor, J.W. (2003) Short-term electricity demand forecasting using double seasonal exponential smoothing. Journal of the Operational Research Society, 54, 799-805., for the Double Seasonal Holt-Winters using dshw in R, but I get different MAPE values.

This is my code for 2 periods-ahead, based on https://otexts.org/fpp2/forecasting-on-training-and-test-sets.html and https://robjhyndman.com/hyndsight/rolling-forecasts/ by Rob J. Hyndman:

   library("forecast")
    train <- msts(taylor[1:2688], seasonal.periods=c(48,336), ts.frequency=48)
    test <- msts(taylor[2689:4032], seasonal.periods=c(48,336), ts.frequency=48, start=57)
    fit <- dshw(train, armethod=FALSE)


  h=2
  n <- length(test) - h + 1
  fc <- ts(numeric(n), start=2688+h)

    for(i in 1:n)
    {  
      x <- msts(taylor[1:2688+i-1], seasonal.periods=c(48,336), ts.frequency=48)
      refit <- dshw(x, model=fit, armethod=FALSE)
      fc[i] <- forecast(refit, h=h)$mean[h]
   }

  mape <- mean(abs(taylor[(2688+h):4032]-fc)/taylor[(2688+h):4032])*100)

Can I do this for h=1,...48 in a for loop and plot the mape values I get or this is not the way to do it?

The MAPE results should be the dots shown in next figure:

Answer 1

Unfortunately, there is no "the" correct way to calculate the MAPE. Your approach uses rolling origins and then evaluates over the entire rest of the test sample, starting at horizon $h$. If this does not yield the same results as in the paper, try calculating the MAPE only at the correct horizon, by only storing a single forecast for the horizon $h$ for each origin, then averaging these single forecasts over the origins.

Answer 2

Here's a summary of what we do on my team. We generate a forecast with a horizon of 10 periods (10 weekdays, the next 2 weeks). We want to do an 8 week holdout. So we run the forecast 8 times, starting 10 weeks ago and then using the 2nd week of our forecast each time as the week we're evaluating. So we end up with 40 periods of holdout data and then compute our metrics on that, where each of those 40 periods was at the same point in terms of steps-ahead (6 to 10 days ahead). There's not a single correct way to do it but I'd advise you think about how the forecast will be used, and then try to evaluate it the same way now as your testing that you would use later to evaluate performance when it is productionized.