Model comparison between an ARIMA model and a regression model

Question

I'm really having trouble finding out how to compare ARIMA and regression models. I understand how to evaluate ARIMA models against each other, and different types of regression models (ie: regression vs dynamic regression with AR errors) against each other, however I cannot see many commonalities between ARIMA model and regression model evaluation metrics.

The only two metrics they share is the SBC & AIC. ARIMA output produces neither a root MSE figure or an r^2 statistic. I'm not too sure whether the standard error estimate of an ARIMA model is directly equivalent (or comparable) to anything within regression outputs.

If anyone could point me in the right direction that would be great, as I'm really confused here. I feel like i'm trying to compare apples with oranges.

I'm using SAS by the way in conducting this analysis.

Answer 1

If we exclude the ARIMAX models, which are ARIMA with regressors, ARIMA and regression models are models with different approaches. ARIMA tries to model the variable only with information about the past values of the same variable. Regression models on the other hand model the variable with the values of other variables. Since these approaches are different, it is natural then that models are not directly comparable.

On the other hand since both models try to model one variable, they both produce the modelled values of this variable. So the question of model comparison is identical to comparison of modelled values to true values. For more information how to do that the seventh chapter of Elements of Statistical Learning by Hastie et al. is an enlightening read.

Update: Note that I do not advocate comparing only in sample fit, just that when models are different the natural way to compare models is to compare their outputs, disregarding how they were obtained.

Answer 2

You could use the MSE/AIC/BIC of the arima model and compare it to the MSE/AIC/BIC of the regression model. Just make sure that the number of fitted values are the same otherwise you might be making a mistake. For example if the ARIMA model has a lag structure of say sp+p ( a seasonal difference of order sp and an autoregressive structure of order p , you lose the first sp+p datapoints and only NOB-SP-P values are actually fit. If the regression model has no lags then you have NOB fitted points or less depending upon your specification of the lagged values for the inputs. So one has to realize that the MSE's may not be on the same historical actual values. One approach would be to compute the MSE of the regression model on the last NOB-SP-P values to put the models on an equal footing. You might want to GOOGLE "regression vs box-jenkins" and get some pointers on this and more. In closing one would normally never just fit a regression model with time series as their may be information in the lags of the causals and the lags of the dependent variable justifying the STEP-UP from regression to a Transfer Function Model a.k.a ARMAX Model . If you didn't STEP-UP then one one or more of the Gauusian Assumptions would be voided making your F/T tests meaningless and irrevelant. Furthermore there may be violations of the constancy of the error term requiring the incorporation of level shifts/local time trends and either pulse or seasonal pulse variable to render the error process having a "mean of 0.0 everywhere"

Answer 3

Cross validation would probably be good here. To do this you split your data set into 2 parts. You use the first part to fit both models, and then use the fitted model to predict the second part. This can be justified as an approximation to a fully Bayesian approach to model selection. We have the likelihood of a model $M_{i}$

$$p(d_{1}d_{2}...d_{N}|M_{i}I)=p(d_{1}|M_{i}I)\times p(d_{2}|d_{1}M_{i}I)\times p(d_{3}|d_{1}d_{2}M_{i}I)\times..$$ $$..\times p(d_{N}|d_{1}d_{2}...d_{N-1}M_{i}I)$$

Which can be seen heuristically as sequence of predictions, and then of learning from mistakes. You predict the first data point with no training. Then you predict the second data point after learning about the model with the first one. Then you predict the 3rd data point after using the first two to learn about the model, and so on. Now if you have a sufficiently large data set, then the parameters of the model will become well determined beyond a certain amount of data, and we will have, for some value $k$:

$$p(d_{k+2}|d_{1}....d_{k}d_{k+1}M_{i}I)\approx p(d_{k+2}|d_{1}....d_{k}M_{i}I)$$

The model can't "learn" any more about the parameters, and is basically just predicting based on the first $k$ observations. So I would choose $k$ (the size of the first group) to be large enough so that you can accurately fit the model, $20$-$30$ data points per parameter is probably enough. You also want to choose $k$ large enough so that the dependence in the $d_{k+1}...d_{N}$ which is being ignored does not make the approximation useless.

Then I would simply evaluate the likelihoods of each prediction, and take their ratio, interpreted as a likelihood ratio. If the ratio is about $1$, then neither model is particularly better than the other. If it is far away from $1$ then this indicates one of the models is outperforming the other. a ratio of under 5 is weak, 10 is strong, 20 very strong, and 100, decisive (corresponding reciprocal for small numbers).