3
$\begingroup$

I'm currently trying to make my first ARIMA model. In my limited experience with linear regressions I find that R-squared gives me a pretty good absolute idea of how the model performs. I haven't found anything similar to that with ARIMA models. I know that there is AIC and SBC, but from my understanding they don't mean much by themselves and are used to compare the goodness of fit to other models. Am I correct in my understanding of AIC and SBC? Is there a metric that is similar to R-squared for ARIMA models in that it is a standalone measure of goodness of fit?

| cite | improve this question | | | | |
$\endgroup$
1
$\begingroup$

$R^2$ is as valid for ARIMA models as it is for linear regression models. In both cases it measures the proportion of the total variance that is explained by the model. It also has the same drawbacks for ARIMA as for linear regression models.

I do not see why goodness of fit should be measured in different ways for ARIMA vs. linear regression, so your question reduces to a more general question of measuring goodness of fit in linear models. That has been discussed in other threads under the tag .

| cite | improve this answer | | | | |
$\endgroup$
  • $\begingroup$ Thank you for the explanation. I did not realize that goodness-of-fit measures for linear regression are valid for ARIMA as well. I made a poor assumption that they were different because SAS outputs different measures of goodness-of-fit for the two procedures. With this new knowledge I can take my study of the subject in the right direction. $\endgroup$ – Jarom Feb 13 '17 at 15:07
  • $\begingroup$ @Downvoter, could you give some constructive feedback so that I can improve the answer? $\endgroup$ – Richard Hardy Dec 25 '19 at 9:36
1
$\begingroup$

The Box-Ljung residuals can be used to assess the fit of the time series model. Of course there are no absolutes and all measures have drawbacks. I am not sure how one would interpret $R^2$ in the context of a time series. Also ARIMA models are not linear models. They are more general. Besides the many threads on this site covering these types of models you can find out a lot about them in the most current edition of Box and Jenkins Time Series Analysis: Forecasting and Control. This book is now in its 5th Edition published in June 2015 by Wiley. Additional coauthors include Gregory C. Reinsel and Greta M. Ljung. A refernce at the Wiley site is : www.wiley.com/WileyTitle/productCd-1118675029.html. In addition the bootstrap approach to time series analysis is covered in a number of books including my book An Introduction to Bootstrap Methods with Applications to R which I coauthored with Robert A. LaBudde and was also published by Wiley in 2011.

| cite | improve this answer | | | | |
$\endgroup$
  • $\begingroup$ The note that ARIMA models are not linear is very interesting. Could you briefly elaborate in what sense you mean that? When expressed in terms of the observable lags and unobservable errors and their lags, the models are linear in parameters. But if we properly deal with the fact that errors are unobservable, then things get more complicated; we have an infinite-order ARMA that can be parsimoniously represented by a ratio of two lag polynomials, which means some parameters are in the denominator - there is the nonlinearity. Or did you mean something else? $\endgroup$ – Richard Hardy Feb 13 '17 at 15:25
  • $\begingroup$ Did you think about this? $\endgroup$ – Richard Hardy Feb 28 '17 at 15:16
0
$\begingroup$

There are typically no "absolutes" when assessing model performance. For a simple and realistic alternative to in-sample measures like BIC, R-squared, etc, try what I call "percent of variation explained," which I define as $1-\frac {\|\epsilon\|}{\|y-m\|}$ where $\epsilon$ are your forecast errors, $y$ is your observed out of sample time series, $\|\cdot\|$ is an appropriately-chosen norm, and $m$ is a measure of center for $y$ according to your choice of norm. This metric will be equal to 1 only if your model perfectly fits $y$.

That means to first split your data into two sets, build your model on the first set, check this metric on the second. Or, you can also try what's known as time series cross validation.

Googling this topics will complete the picture if it sounds at all vague.

| cite | improve this answer | | | | |
$\endgroup$
  • 2
    $\begingroup$ I wonder how you suggest to do $k$-cross validation in the time series setting. Rolling windows is the more common alternative. Also, $R^2$ is not a sensible measure of out-of-sample fit because it completely neglects forecast bias (nonzero mean of forecast errors) and only accounts for the variance of forecast errors. You can easily construct an example to show that. $\endgroup$ – Richard Hardy Feb 11 '17 at 9:33
  • $\begingroup$ Great comments. When I say $k$-fold in a time series context, I do mean the rolling window approach with $k$ windows. You're also correct about $R^2$, rather, I should say out of sample MSE or something like $1-\|\epsilon\|/\|y\|$ where $\epsilon$ is forecast residuals and $y$ is the observed time series. $\endgroup$ – Mustafa S Eisa Feb 11 '17 at 10:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.