2
$\begingroup$

I'm writing a blog post on forecasting time series with autoregression. In it, I compare the performance of SLR, ARIMA, and SARIMAX on forecasting the number of Home Sales in Seattle (see below).

All 3 have different numbers of "input parameters": SLR just uses time, ARIMA and SARIMAX both use time and 12 lagged $y$ values. *I say "input parameters" b/c I'm not sure how to consider $y$.

I'm currently using RMSE to compare them. Is this an acceptable practice, or is there another measure I should use that takes model complexity into account (e.g. something akin to adjusted R^2)?

I know that MAPE is a commonly used forecasting metric. But like RMSE, I'm not sure it's appropriate for comparing models with different numbers of input parameters. Just wondering if there's anything better out there.

enter image description here

Improve this question
$\endgroup$
5
  • 1
    $\begingroup$ Are you measuring RMSE in-sample or out-of-sample? $\endgroup$
    – Chris Haug
    Commented Sep 3, 2018 at 21:21
  • $\begingroup$ Out of sample. I'm interested in the RMSE of the forecast, plus the RMSE of the in-sample ARIMA and SARIMAX models are nearly identical (the differences b/t them only really show out-of-sample). $\endgroup$
    – infinitely_improbable
    Commented Sep 4, 2018 at 4:27
  • $\begingroup$ Rather than choosing a single measure, it is more common to include several relevant measures of accuracy for prediction. $\endgroup$
    – Frans Rodenburg
    Commented Sep 4, 2018 at 8:14
  • 3
    $\begingroup$ These kinds of adjustments for model complexity are typical of in-sample measures, but not of out-of-sample measures, which tackle overfitting in an entirely different way. You can compare RMSE/MAPE/MASE/etc out-of-sample between these models, just make sure that your exogenous regressors aren't assumed known in the future if they aren't really. $\endgroup$
    – Chris Haug
    Commented Sep 4, 2018 at 12:18
  • $\begingroup$ @ChrisHaug: do you want to post your comment as an answer? In addition, the forecasting textbooks listed at this thread all give error metrics commonly used in forecast evaluation. OP may be interested in the shortcomings of the MAPE. And $R^2$ is not commonly used in forecast evaluation. $\endgroup$
    – Stephan Kolassa
    Commented Dec 8, 2023 at 23:23

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.