1
$\begingroup$

I have time-series data from 1985 Q1 until 2010 Q4. Then, the time series data have been split into training and test set. The split is as follows

  1. In-sample set from 1985 Q1 until 2004 Q4
  2. Out-sample set - (a) 2005 Q1 until 2005 Q4, and (b) 2005 Q1 until 2010 Q4

After that, the forecast for (a) the period of 2005 Q1 until 2005 Q4 and (b) the period of 2005 Q1 until 2010 Q4 have been produced using the ETS exponential smoothing method and the forecast accuracy is measured. Let say the forecast error measures for the test set are as in the table below, what we can say because MAE decreases but MAPE increases? Can we compare the error measures of MAE and MAPE of Forecast (a) and Forecast (b) even though the length of the forecast period are different? I am new to forecasting which makes me incompetent in this issue.

Forecast MAE MAPE Theil's U
(a) 2005Q1-2005Q4 72.75 68.25 0.62
(b) 2005Q1-2010Q4 64.96 74.47 0.79
Improve this question
$\endgroup$

1 Answer 1

Reset to default
0
$\begingroup$

You can in principle compare MAEs and MAPEs between the two forecast periods, as in "one is smaller than the other". I would be very careful about interpreting this. Statistical tests like the Diebold-Mariano tests usually presuppose that the forecasts being evaluated are calculated based on the same underlying actuals (though the DM test may well be robust to this).

I would definitely look at the underlying series and try to understand whether there was a change in the underlying data generating process.

Your MAPEs are quite large, which to me indicates a potentially high coefficient of variation of your holdout sample. In such circumstances, the difference between the conditional expectation (which is what ETS aims at) and the conditional (-1)-median (which is what the MAPE aims at) becomes relevant. If this is gibberish to you, then you may want to take a look at What are the shortcomings of the Mean Absolute Percentage Error (MAPE)? and the literature cited there.

Improve this answer
$\endgroup$
3
  • $\begingroup$ By comparing the MAEs and MAPEs between two forecast periods, can it be interpreted as "the model produces less error in the long-run forecast, and therefore, the model is better at forecasting for the long run? $\endgroup$
    – Anonymous M
    Apr 22, 2021 at 12:06
  • 1
    $\begingroup$ Well, you can say that the model produced a lower error for one forecast period than for another one. If a lower MAPE is your definition of a better forecast, then that's a better forecast. Whether something similar will continue to hold is not obvious. $\endgroup$
    – Stephan Kolassa
    Apr 22, 2021 at 12:37
  • $\begingroup$ I run various models to predict two years in the future choosing from a hold out sample based on MAPE. Just because one model worked well, does not mean it will work well in the future. If it did time series would be easy. Structural breaks, pulses, change in policy, a million things can make a difference. $\endgroup$
    – user54285
    Apr 23, 2021 at 21:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.