I have two forecasting models, moving average and single exponential smoothing. The values of Mean Absolute Percentage Error (MAPE) is 5.2%, 5.8%. Since the difference of MAPE between the models are very close, I am quite confused which model to choose. Can we perform a t-test or ANOVA on Absolute Percentage Error (APE) data sets of two different models to check whether the difference is significant? If the difference is significant can we choose the model with low MAPE?
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
As you write, any two forecasting methods will yield slightly different forecasts and therefore be slightly differently accurate. The question is whether the difference in accuracy could have occurred by chance - that is, whether the difference is statistically significant.
The standard test here is to collect many errors from your two forecasting methods (e.g., absolute or squared errors), take the mean of differences and standardize, then compare to a $N(0,1)$ distribution. This technique is known as the Diebold-Mariano test. It should be available in many forecasting software packages.
The original publication ws Diebold & Mariano (1995, Journal of Business and Economic Statistics). Twenty years later, Diebold (2015, Journal of Business and Economic Statistics) looked back.
answered Apr 16, 2016 at 15:07