I'm trying to get the same results reported in the paper Taylor, J.W. (2003) Short-term electricity demand forecasting using double seasonal exponential smoothing. Journal of the Operational Research Society, 54, 799-805., for the Double Seasonal Holt-Winters using dshw in R, but I get different MAPE values.

This is my code for 2 periods-ahead, based on https://otexts.org/fpp2/forecasting-on-training-and-test-sets.html and https://robjhyndman.com/hyndsight/rolling-forecasts/ by Rob J. Hyndman:

    train <- msts(taylor[1:2688], seasonal.periods=c(48,336), ts.frequency=48)
    test <- msts(taylor[2689:4032], seasonal.periods=c(48,336), ts.frequency=48, start=57)
    fit <- dshw(train, armethod=FALSE)

  n <- length(test) - h + 1
  fc <- ts(numeric(n), start=2688+h)

    for(i in 1:n)
      x <- msts(taylor[1:2688+i-1], seasonal.periods=c(48,336), ts.frequency=48)
      refit <- dshw(x, model=fit, armethod=FALSE)
      fc[i] <- forecast(refit, h=h)$mean[h]

  mape <- mean(abs(taylor[(2688+h):4032]-fc)/taylor[(2688+h):4032])*100)

Can I do this for h=1,...48 in a for loop and plot the mape values I get or this is not the way to do it?

The MAPE results should be the dots shown in next figure:

enter image description here

  • $\begingroup$ What is your question? You lead in with getting different MAPEs, but don't give details, and later, you ask whether you can do this (of course you can, if it is what you want). Can you please clarify? $\endgroup$ – Stephan Kolassa Aug 11 '18 at 11:11
  • $\begingroup$ I am wondering if this is the correct form to calculate the MAPEs for different horizonts or if I have to do it in a different way to get the proper result (the MAPE of the test set in h-ahead periods using the model dshw in the training set) because I am not completely sure what I obtain doing this $\endgroup$ – Iciar Aug 11 '18 at 11:26

Unfortunately, there is no "the" correct way to calculate the MAPE. Your approach uses rolling origins and then evaluates over the entire rest of the test sample, starting at horizon $h$. If this does not yield the same results as in the paper, try calculating the MAPE only at the correct horizon, by only storing a single forecast for the horizon $h$ for each origin, then averaging these single forecasts over the origins.


Here's a summary of what we do on my team. We generate a forecast with a horizon of 10 periods (10 weekdays, the next 2 weeks). We want to do an 8 week holdout. So we run the forecast 8 times, starting 10 weeks ago and then using the 2nd week of our forecast each time as the week we're evaluating. So we end up with 40 periods of holdout data and then compute our metrics on that, where each of those 40 periods was at the same point in terms of steps-ahead (6 to 10 days ahead). There's not a single correct way to do it but I'd advise you think about how the forecast will be used, and then try to evaluate it the same way now as your testing that you would use later to evaluate performance when it is productionized.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.