# Accuracy metric for comparing Time Series models?

I'm writing a blog post on forecasting time series with autoregression. In it, I compare the performance of SLR, ARIMA, and SARIMAX on forecasting the number of Home Sales in Seattle (see below).

All 3 have different numbers of "input parameters": SLR just uses time, ARIMA and SARIMAX both use time and 12 lagged $y$ values. *I say "input parameters" b/c I'm not sure how to consider $y$.

I'm currently using RMSE to compare them. Is this an acceptable practice, or is there another measure I should use that takes model complexity into account (e.g. something akin to adjusted R^2)?

I know that MAPE is a commonly used forecasting metric. But like RMSE, I'm not sure it's appropriate for comparing models with different numbers of input parameters. Just wondering if there's anything better out there.

• Are you measuring RMSE in-sample or out-of-sample? – Chris Haug Sep 3 '18 at 21:21
• Out of sample. I'm interested in the RMSE of the forecast, plus the RMSE of the in-sample ARIMA and SARIMAX models are nearly identical (the differences b/t them only really show out-of-sample). – infinitely_improbable Sep 4 '18 at 4:27
• Rather than choosing a single measure, it is more common to include several relevant measures of accuracy for prediction. – Frans Rodenburg Sep 4 '18 at 8:14
• These kinds of adjustments for model complexity are typical of in-sample measures, but not of out-of-sample measures, which tackle overfitting in an entirely different way. You can compare RMSE/MAPE/MASE/etc out-of-sample between these models, just make sure that your exogenous regressors aren't assumed known in the future if they aren't really. – Chris Haug Sep 4 '18 at 12:18