15
$\begingroup$

I've been struggling to find meaningful guidelines for comparing models based on differences in AIC. I keep coming back to the rule of thumb offered by Burnham & Anderson 2004, pp. 270-272:

Some simple rules of thumb are often useful in assessing the relative merits of models in the set: Models having $\Delta i$ ≤ 2 have substantial support (evidence), those in which 4 ≤ $\Delta i$ ≤ 7 have considerably less support, and models having $\Delta i$ > 10 have essentially no support.

For example, see these questions:

I'm trying to understand the rational justification for these threshold numbers of 2, 4, 7 and 10. The Wikipedia article on Akaike Information Criterion gives some guidance on this question:

Suppose that there are R candidate models. Denote the AIC values of those models by AIC1, AIC2, AIC3, …, AICR. Let AICmin be the minimum of those values. Then the quantity exp((AICmin − AICi)/2) can be interpreted as being proportional to the probability that the ith model minimizes the (estimated) information loss.

As an example, suppose that there are three candidate models, whose AIC values are 100, 102, and 110. Then the second model is exp((100 − 102)/2) = 0.368 times as probable as the first model to minimize the information loss. Similarly, the third model is exp((100 − 110)/2) = 0.007 times as probable as the first model to minimize the information loss.

So, based on this information, I compiled the following table:

Delta AIC probabilities

  • delta_aic: Minimum AIC of all the models being compared minus AIC of the i th model under consideration. In the Wikipedia example, delta_aic for the model where AIC=100 is 0, for the model where AIC=102 is 2, and for the model where AIC=110 is 10.
  • prob(min vs. i): Model i is x times as probable as the minimum model to minimize the information loss. In the Wikipedia example, the model where AIC=100 is equally probable, the model where AIC=102 is 0.368 times as probable, and the model where AIC=110 is 0.007 times as probable.
  • prob(i vs. min): This is simply 1/prob(min vs. i). So, it means that the minimum model is x times as probable as Model i to minimize the information loss. In the Wikipedia example, the model where AIC=100 is equally probable, the minimal model is 2.7 times as probable as the model where AIC=102, and the minimal model is 148.4 times as probable as the model where AIC=110.

OK, so I think I get the mathematics somewhat, but I really don't understand what all this means in terms of practically selecting one model over another. The rule of thumb says that if the minimal model is 2.7 times as probable as another model (that is, $\Delta i$ ≤ 2), then the two models are practically equivalent. But why is 2.7 times probability in minimizing information so little as to be of no consequence? What does that even mean? Similarly, the rule of thumb says that only when you get to when the minimal model is 148.4 times as probable as another model (that is, $\Delta i$ > 10) would you say that the model under consideration can no longer be considered equivalent in any meaningful way. Isn't that an extremely huge amount of tolerance?

Even when breaking it down mathematically this way, the rule of thumb still does not make any intuitive sense to me. So, this brings me to my question:

  • Could someone please explain a simple logical justification for the commonly accepted rule of thumb for acceptable AIC differences?
  • Alternatively, could someone please provide alternative rational criteria that are better justified than this rule of thumb?
$\endgroup$
8
  • 2
    $\begingroup$ I don't want to make it an answer 'cause it's only my guess, that doesn't fit perfectly to the numbers: $1-\exp(-\frac{1}{2}\cdot 2)\approx0.63$, which is quite close to the $1\sigma$ level (see the 68-95-99.7 rule). Similarly, $1-\exp(-\frac{1}{2}\cdot 6)\approx0.95$ ($2\sigma$) and $1-\exp(-\frac{1}{2}\cdot 10)\approx0.993$ ($3\sigma$). Maybe the first threshold (2) was chosen to be a round number; maybe $7$ was chosen instead $6$ 'cause it's a lucky number; maybe $10$ was chosen 'cause it's a decade, so a very round number. $\endgroup$ – corey979 Jun 5 '18 at 18:57
  • 1
    $\begingroup$ @corey979, but what does $1−exp(−12⋅2)≈0.63$ actually mean, in layman's terms? Is it that 63% of models with the same real amount of information (in the sense of Akaike "Information" Criterion) would produce AIC values within that range? That would make sense, if that is what it means. (I'm not a statistician; I'm just an advanced user of statististics that tries to understand the real-world meaning behind the procedures I use.) $\endgroup$ – Tripartio Jun 7 '18 at 10:29
  • 2
    $\begingroup$ I do not know about the rules of thumb, but I would caution against the use of any critical cutoff that entails a (or substitutes for) a decision. A decision regarding the merits of models should be made on the basis of the AIC and all other information. $\endgroup$ – Michael Lew May 17 at 20:57
  • 1
    $\begingroup$ It is correct to argue against such rules of thumb. The log likelihood scale, while extremely useful, is proportional to the sample size. A difference in AIC of 1000 can be inconsequential for n=1,000,000. You may not be able to notice improved predictions from the one with lower AIC in this case. Need to supplement AIC with relative likelihood ratio $\chi^2$ and with other sample-size independent measures such as median absolute prediction error. See also here. $\endgroup$ – Frank Harrell Jun 23 at 12:10
  • 1
    $\begingroup$ Sorry correct link is fharrell.com/post/addvalue $\endgroup$ – Frank Harrell Jun 24 at 15:03
1
$\begingroup$

I encountered the same issue, and was trying to search an answer in related articles. The Burnham & Anderson 2002 (Model Selection and Multimodel Inference - A Practical Information-Theoretic Approach Second Edition) book actually used three approaches to derive these empirical numbers. As stated in Chapter 4.5 in this book, one approach (which is easiest to understand), is let $\Delta_p = AIC_{best} - AIC_{min}$ be a random variable with a sampling distribution. They have done Monte Carlo simulation studies on this variable, and the sampling distribution of this $\Delta_p$ has substantial stability and the 95th percentile of the sampling distribution of $\Delta_p$ is generally much less than 10, and in fact generally less than 7 (often closer to 4 in simple situations), as long as when observations are independent, sample sizes are large, and models are nested.

$\Delta_p > 10$ is way beyond the 95th percentile, and is thus highly unlikely to be the Kullback-Leibler best model.

In addition, they actually argued against using 2 as a rule of thumb in their Burnham, Anderson, and Huyvaert et al., 2011 paper. They said $\Delta$ in the 2-7 range have some support and should rarely be dismissed.

New contributor
huajun is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct.
$\endgroup$
0
$\begingroup$

I might be able to provide some justification for the cutoff for AICs less than 2 units apart. I wrote a paper analyzing Quetelet's famous analysis of 5723 Scottish chest girths, one of the first applications of what would come to be called the normal distribution. Quetelet, long before goodness of fit tests, argued that the chest data were normal. Others have disagreed. The AIC for the fit of the normal to Quetelet's Scottish chest data is 24629. I generated 10000 random data with n = 5732 using Matlab's pseudo-random normal generator with the same mean and sd as Quetelet's data, obtaining a mean AIC of 24630 ± 2 [± half 95% confidence interval]. I would certainly agree with the ▵AIC = 2 cutoff, but I have no idea about justification for the 4-7 or >10 cutoffs. Gallagher, E. D. (2020). Was Quetelet's Average Man Normal? The American Statistician/Taylor & Francis, 74(3), 301-306. https://doi.org/10.1080/00031305.2019.1706635.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.