2
$\begingroup$

I'm writing a blog post on forecasting time series with autoregression. In it, I compare the performance of SLR, ARIMA, and SARIMAX on forecasting the number of Home Sales in Seattle (see below).

All 3 have different numbers of "input parameters": SLR just uses time, ARIMA and SARIMAX both use time and 12 lagged $y$ values. *I say "input parameters" b/c I'm not sure how to consider $y$.

I'm currently using RMSE to compare them. Is this an acceptable practice, or is there another measure I should use that takes model complexity into account (e.g. something akin to adjusted R^2)?

I know that MAPE is a commonly used forecasting metric. But like RMSE, I'm not sure it's appropriate for comparing models with different numbers of input parameters. Just wondering if there's anything better out there.

enter image description here

$\endgroup$
  • 1
    $\begingroup$ Are you measuring RMSE in-sample or out-of-sample? $\endgroup$ – Chris Haug Sep 3 '18 at 21:21
  • $\begingroup$ Out of sample. I'm interested in the RMSE of the forecast, plus the RMSE of the in-sample ARIMA and SARIMAX models are nearly identical (the differences b/t them only really show out-of-sample). $\endgroup$ – infinitely_improbable Sep 4 '18 at 4:27
  • $\begingroup$ Rather than choosing a single measure, it is more common to include several relevant measures of accuracy for prediction. $\endgroup$ – Frans Rodenburg Sep 4 '18 at 8:14
  • 2
    $\begingroup$ These kinds of adjustments for model complexity are typical of in-sample measures, but not of out-of-sample measures, which tackle overfitting in an entirely different way. You can compare RMSE/MAPE/MASE/etc out-of-sample between these models, just make sure that your exogenous regressors aren't assumed known in the future if they aren't really. $\endgroup$ – Chris Haug Sep 4 '18 at 12:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.