3
$\begingroup$

I have two forecasting models, moving average and single exponential smoothing. The values of Mean Absolute Percentage Error (MAPE) is 5.2%, 5.8%. Since the difference of MAPE between the models are very close, I am quite confused which model to choose. Can we perform a t-test or ANOVA on Absolute Percentage Error (APE) data sets of two different models to check whether the difference is significant? If the difference is significant can we choose the model with low MAPE?

$\endgroup$
1
$\begingroup$

As you write, any two forecasting methods will yield slightly different forecasts and therefore be slightly differently accurate. The question is whether the difference in accuracy could have occurred by chance - that is, whether the difference is statistically significant.

The standard test here is to collect many errors from your two forecasting methods (e.g., absolute or squared errors), take the mean of differences and standardize, then compare to a $N(0,1)$ distribution. This technique is known as the Diebold-Mariano test. It should be available in many forecasting software packages.

The original publication ws Diebold & Mariano (1995, Journal of Business and Economic Statistics). Twenty years later, Diebold (2015, Journal of Business and Economic Statistics) looked back.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.