4
$\begingroup$

When building a model for a time series, I have received different recommendations -

One is to choose the model with the smallest AIC_C value, e.g. do an initial ACF/PACF analysis of the raw data to get an idea of what kind of model to go for, and then choose between candidate models by going for the one that minimises the AIC_C value. This approach is described here.

Another is to choose the model with the nicest ACF and PACF (i.e. no significant spikes, not with any obvious structure and following a normal distribution). In this approach, remove one kind of variance at a time, look at the ACF/PACF again, remove another kind of variance based on that and keep going until the ACF/PACF can't be improved any further. Use likelihood ratio tests to decide between models if the eyes can't detect the difference.

I have seen many examples where the one of these approaches chooses one model and the other chooses another model.

Is it possible to generally say which is better?

(I'm using R for analysis if that's of any interest)

EDIT: In my present analysis, the AICc approach chooses an ARIMA(1,0,0)(0,1,0)[12] model, whereas the ACF/PACF approach chooses an ARIMA(0,1,0)(1,0,0)[12] model. The AICc model has an AICc of 307.21 and a standard error of 0.142, whereas the ACF/PACF model has an AICc of 463.99 and a standard error of 0.0704.

Improve this question
$\endgroup$
0

2 Answers 2

Reset to default
1
$\begingroup$

The problem with AIC-based relationships is that they premise : 1) no pulses/level shifts/seasonal pulses/local time trends exist in the data because if they did ( and they always do) the AIC statistics are meaningless because they premise a model free of that structure; 2) that there is constant error variance and constant parameters in the premised model: and 3) that there are no missing lags in the premised model. These assumptions do not hold for a data-based approach as the Gaussian Violations can all be tested and rectified by possible model augmentation or simplification. The whole idea of empirically identifying a useful model is to follow the concepts of the General Linear Model in forming both memory and any needed deterministic structure while encoding any needed transformations such as power transforms/weighted least squares and arch/garch .

the big lie in econometrics is that you are supposed to know/guess your model before you observed the data. This of course could lead to throwing out the observed data until you find data that conforms to your pre-specified model. … just teasing here. The econometrician is not supposed to discover new structure but in effect when you delete specified structure you are changing the model. The pure empiricist starts with no model and carefully both augments and refines the current candidate until all coefficients are statistically significant and that there is no evidented structure in the model’s residuals. The software that I use allows not only the pure theoretical specification and the pure empirical approach BUT the middle ground where a pre-specified/theoretical model is used as the Starting Model

A reasonable place to start are the flow diagrams that I prepared using sound exploratory techniques http://www.autobox.com/cms/index.php/blog/entry/build-or-make-your-own-arima-forecasting-model in response to a query much like the one from @Richard Hardy.

Improve this answer
$\endgroup$
7
  • $\begingroup$ I appreciate the idea of building a model in the way you suggest. I also appreciate the idea of guessing the model before the data is observed. There is a theoretical justification for that, too, even though it might be an inferior approach in practice. But say I built a model your way and discovered that AIC-wise it is inferior to a simpler model which does not account for all the pulses, local time trends and the like. I would interpret this as evidence of overfitting while trying to account for all the features observed in the data. Would you suggest to stick to the model regardless? $\endgroup$
    – Richard Hardy
    Commented Feb 19, 2015 at 20:23
  • $\begingroup$ Also, I do not think that selecting a model using AIC means we are necessarily choosing from a pool of "stupid" models. Suppose I built two models following your suggestions and they are quite similar but differ in small details due to borderline cases where it is difficult to determine whether there is a pulse or not, for example. I would like to use AIC to choose one of those models. I guess many would agree that this is a reasonable use of AIC. But what if I had a hundred models built following your suggestions and differing only in small details; would the use of AIC be justified now? $\endgroup$
    – Richard Hardy
    Commented Feb 19, 2015 at 20:27
  • $\begingroup$ Finally, the points 1), 2) and 3) do not automatically come with AIC. They may rather arise from a "stupid" pool of candidate models, and I agree that this may often be a problem. But it is not due to the use of AIC per se. Thus I think you are criticizing a different thing rather than the more general idea of "model selection by AIC". (Frankly speaking, I must agree with basically everything you say about model building; I think we just fail to understand what object exactly one is defending or criticizing.) $\endgroup$
    – Richard Hardy
    Commented Feb 19, 2015 at 20:31
  • $\begingroup$ "I would interpret this as evidence of overfitting while trying to account for all the features observed in the data. Would you suggest to stick to the model regardless? – Richard Hardy 30 mins ago" from tour comment. It probably would not be over-fitting BUT it could be due to smaller standard errors based upon reduced error sums of squares thus falsely evidenting significance. Care should be taken either way. Please contact me via email and we can continue this dialogue as I think we have run out of space !. $\endgroup$
    – IrishStat
    Commented Feb 19, 2015 at 20:55
  • $\begingroup$ I should add that my last comments dealt with small sample sizes $\endgroup$
    – IrishStat
    Commented Feb 19, 2015 at 21:03
0
$\begingroup$

It's a pretty general question and any simple answer will not be completely satisfactory. However...

If you have a number of candidate models, choose by AICc. This will give you the best model (in a certain sense, defined by the AICc criterion) available given the data sample you have. Even though noise may dominate over the signal in your data (which is a popular criticism), still AICc will give you the best there is in terms of this particular criterion.

Meanwhile, choosing by ACF/PACF would mean data mining. Choosing a model from a pool of candidate models by looking for a desired feature (like a nice ACF/PACF plot) is not the same as first choosing a model and then checking whether it has the desired feature. It is a bit like saying "I will toss a coin; if it comes heads up, I will go do the washing up; if tails, I will watch TV. <...toss...> Oops, it came heads up... I better try again... Oh no, heads up again! Try once more... Hooray, now it's tails! I can go watch TV!" You only trick yourself this way.

Improve this answer
$\endgroup$
7
  • $\begingroup$ If one has a prior based upon sound economic thinking then by all means it should be tested and evaluated as to whether it works. By working we mean statistically significant coefficients and a set of residuals which are free of both stochastic and deterministic structure and of course has constant parameters and error variance for all sub-intervals of time. In the absence of that, using sound exploratory techniques not list-based like AIC or BIC one may actually find a model that separates signal from noise. Remember all models are wrong but some are useful (ascribed to G.E.P. Box). $\endgroup$
    – IrishStat
    Commented Feb 18, 2015 at 21:33
  • $\begingroup$ @IrishStat, I often get the remark that AIC-based selection suffers from noise in the data. I do not oppose that. However, I believe this holds for any data-based technique so that this remark is not very helpful in practice when making modelling choices. I have asked before (in comments to similar discussions) what could be a better choice than using AIC but never got an answer... Could you please indicate what kind of sound exploratory techniques that are capable of separating signal from noise you have in mind? $\endgroup$
    – Richard Hardy
    Commented Feb 19, 2015 at 6:18
  • $\begingroup$ @IrishStat, Also, I think there will be a difference between (1) when we have a pool of prespecified models we have to choose from and (2) when we are building a model from scratch. My answer was tailored to (1) as I thought this was what the OP was referring to. Would the sound exploratory techniques apply to both (1) and (2) or just (2)? $\endgroup$
    – Richard Hardy
    Commented Feb 19, 2015 at 6:21
  • $\begingroup$ @RichardHardy. Also you can split case 2 to 2a and 2b. 2a) statistician visually observes the data. here brain is clearly fitting a model and then this becomes your prior hypothesis. 2b) the experiment has known certain physical properties. When it comes to 2a, I say AI technology is advancing too rapidly these days that soon we should stop applying human judgement. AGI systems(strong AI) are capable of not only fiddling with parameters of a distribution but also try different distribution functionals. 2b is useful no matter what as it reduces number of samples required even for AI. $\endgroup$
    – Cagdas Ozgenc
    Commented Feb 19, 2015 at 7:11
  • $\begingroup$ I was thinking of the situation in which you build a model from scratch. There is a significant difference in the two approaches that I have been recommended, although they both start with looking at the ACF/PACF patterns of the raw data in order to choose a model to start from. But from there one approach is to go for the models minimising AIC_C, the other is to go for models making the ACF/PACF nicer. And these two approaches choose different models most of the time! $\endgroup$
    – SiKiHe
    Commented Feb 19, 2015 at 9:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.