Testing the difference in AIC of two non-nested models

The whole point of AIC or any other information criterion is that less is better. So if I have two models M1: y = a0 + XA + e and M2: y = b0 + ZB + u, and if the AIC of the first (A1) is less than that of the second (A2), then M1 has a better fit from the information theory standpoint. But is there any cutoff benchmark for the difference A1-A2? How much less is actually less? In other words, is there a test for (A1-A2) other than just eyeballing?

Edit: Peter/Dmitrij... Thanks for responding. Actually, this is a case where my substantive expertise is conflicting with my statistical expertise. Essentially, the problem is NOT choosing between two models, but in checking if two variables which I know to be largely equivalent add equivalent amounts of information (Actually, one variable in the first model and a vector in the second. Think about the case of a bunch of variables as against an index of them.). As Dmitrij pointed out, the best bet seems to be the Cox Test. But is there a way of actually testing the difference between the information contents of the two models?

• You may also be interested in checking out Wagonmakers et al. (2004). Assessing model mimicry using the parametric bootstrap. Journal of Mathematical Psychology, 48, 28-50. (pdf). Jul 19 '12 at 4:22

Is the question of curiosity, i.e. you are not satisfied by my answer here ? If not...

The further investigation of this tricky question showed that there do exist a commonly used rule-of-thumb, that states two models are indistinguishable by $$AIC$$ criterion if the difference $$|AIC_1 - AIC_2| < 2$$. The same you actually will read in wikipedia's article on $$AIC$$ (note the link is clickable!). Just for those who do not click the links:

$$AIC$$ estimates relative support for a model. To apply this in practice, we start with a set of candidate models, and then find the models' corresponding $$AIC$$ values. Next, identify the minimum $$AIC$$ value. The selection of a model can then be made as follows.

As a rough rule of thumb, models having their $$AIC$$ within $$1–2$$ of the minimum have substantial support and should receive consideration in making inferences. Models having their $$AIC$$ within about $$4–7$$ of the minimum have considerably less support, while models with their $$AIC > 10$$ above the minimum have either essentially no support and might be omitted from further consideration or at least fail to explain some substantial structural variation in the data.

A more general approach is as follows...

Denote the $$AIC$$ values of the candidate models by $$AIC1$$, $$AIC2, AIC3, \ldots, AICR$$. Let $$AICmin$$ denotes the minimum of those values. Then $$e^{(AICmin−AICi)/2}$$ can be interpreted as the relative probability that the $$i$$-th model minimizes the (expected estimated) information loss.

As an example, suppose that there were three models in the candidate set, with $$AIC$$ values $$100$$, $$102$$, and $$110$$. Then the second model is $$e^{(100−102)/2} = 0.368$$ times as probable as the first model to minimize the information loss, and the third model is $$e^{(100−110)/2} = 0.007$$ times as probable as the first model to minimize the information loss. In this case, we might omit the third model from further consideration and take a weighted average of the first two models, with weights $$1$$ and $$0.368$$, respectively. Statistical inference would then be based on the weighted multimodel.

Nice explanation and useful suggestions, in my opinion. Just don't be afraid of reading what is clickable!

In addition, note once more, $$AIC$$ is less preferable for large-scale data sets. In addition to $$BIC$$ you may find useful to apply bias-corrected version of $$AIC$$ criterion $$AICc$$ (you may use this R code or use the formula $$AICc = AIC + \frac{2p(p+1)}{n-p-1}$$, where $$p$$ is the number of estimated parameters). Rule-of-thumb will be the same though.

• Hi Dmitrij... I knew you'd spot this. Actually, your response to the original question set this train rolling. I thought this would make an interesting stand-alone question. The problem I'm grappling with is that statistical tests (including Cox's Test) are frequentist and so you can test the differences between two models at some predefined level of significance. But AIC/BIC are likelihood based, and it seems to me that the numbers cannot be directly compared except, as you point out, by rule of thumb. Since IC measures are scale-dependent, an absolute value (2) can be problematic, no? Mar 21 '11 at 6:16
• @user, The absolute value of $2$ is not problematic. You may go for relative probability suggestion, so you will be probably more confident with this than some nice value of $2$. By scale effect you mean when the criterion is less biased in small samples and consistent in large? Try consistent $BIC$ instead and $AICc$ for small samples will be also a good alternative. Rule of thumbs are still usable. Mar 21 '11 at 6:35
• @DmitrijCelov (+1 some time ago) nice answer -- thanks for pasting the text, as Wikipedia no longer has the points covered in the first two paragraphs. The removed paragraph was cited as p. 446: Burnham, K. P., and Anderson, D.R. (2002). Model Selection and Multimodel Inference: A Practical Information-Theoretic Approach, 2nd ed. Springer-Verlag. ISBN 0-387-95364-7.  and the pre-revision Wiki page is here Apr 27 '13 at 23:57
• I should note that I haven't read the Burnham book, and that the old Wiki reference suggested the text as quoted was a paraphrase. FYI, the Wiki page was edited at 16:52, 15 April 2011. Apr 27 '13 at 23:59
• Could you perhaps help with this follow-up question? stats.stackexchange.com/questions/349883/… Jun 5 '18 at 12:12

I think this may be an attempt to get what you don't really want.

Model selection is not a science. Except in rare circumstances, there is no one perfect model, or even one "true" model; there is rarely even one "best" model. Discussions of AIC vs. AICc vs BIC vs. SBC vs. whatever leave me somewhat nonplussed. I think the idea is to get some GOOD models. You then choose among them based on a combination of substantive expertise and statistical ideas. If you have no substantive expertise (rarely the case; much more rarely than most people suppose) then choose the lowest AIC (or AICc or whatever). But you usually DO have some expertise - else why are you investigating these particular variables?

• +1 for emphasizing the need for both statistical and substantive expertise.
– chl
Mar 21 '11 at 10:26