I developed many forecasting models over the same dataset (multiple iteration of simulated time series data). My dataset basically is a multivariate timeseries so the forecasting models forecast many values at the same time. My question is how can I decide which forecasting model is better. I already have calculated the MSE and MAE for each model, but I want to compare them statically. I read about Diebold Mariano test but I think it used for comparing forecasts not models. I also thought of t-test for dependent data but am not sure how can I apply it in my case where I have many models (more than two)

I appreciate your help.

  • 1
    $\begingroup$ If your end goal is forecasting, why isn't it enough to compare the models' forecasts? Models can differ and still give forecasts that are comparably good. $\endgroup$ Commented Apr 8, 2020 at 11:44
  • $\begingroup$ Thank you for your prompt reply. I am just wondering that if I had 10 iterations of the simulation which create slightly different times-series datasets, then I apply different forecasting models (let's say 15 machine-learning models) on these datasets then I took the average of forecasting errors for each model. How can I compare them statically? I know that Diebold Mariano deals with raw errors not (MSE or MAE) In my case, the performance of each model (15 models) represents the average of performance on 10 datasets (and the performance itself represents multiple outputs) $\endgroup$
    – Manal
    Commented Apr 8, 2020 at 13:28
  • $\begingroup$ Regarding comparing many models with something like a Diebold-Mariano test, consult the tag description for diebold-mariano-test. $\endgroup$ Commented Apr 26, 2023 at 17:31


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.