9
$\begingroup$

I've been struggling to find meaningful guidelines for comparing models based on differences in AIC. I keep coming back to the rule of thumb offered by Burnham & Anderson 2004, pp. 270-272:

Some simple rules of thumb are often useful in assessing the relative merits of models in the set: Models having $\Delta i$ ≤ 2 have substantial support (evidence), those in which 4 ≤ $\Delta i$ ≤ 7 have considerably less support, and models having $\Delta i$ > 10 have essentially no support.

For example, see these questions:

I'm trying to understand the rational justification for these threshold numbers of 2, 4, 7 and 10. The Wikipedia article on Akaike Information Criterion gives some guidance on this question:

Suppose that there are R candidate models. Denote the AIC values of those models by AIC1, AIC2, AIC3, …, AICR. Let AICmin be the minimum of those values. Then the quantity exp((AICmin − AICi)/2) can be interpreted as being proportional to the probability that the ith model minimizes the (estimated) information loss.

As an example, suppose that there are three candidate models, whose AIC values are 100, 102, and 110. Then the second model is exp((100 − 102)/2) = 0.368 times as probable as the first model to minimize the information loss. Similarly, the third model is exp((100 − 110)/2) = 0.007 times as probable as the first model to minimize the information loss.

So, based on this information, I compiled the following table:

Delta AIC probabilities

  • delta_aic: Minimum AIC of all the models being compared minus AIC of the i th model under consideration. In the Wikipedia example, delta_aic for the model where AIC=100 is 0, for the model where AIC=102 is 2, and for the model where AIC=110 is 10.
  • prob(min vs. i): Model i is x times as probable as the minimum model to minimize the information loss. In the Wikipedia example, the model where AIC=100 is equally probable, the model where AIC=102 is 0.368 times as probable, and the model where AIC=110 is 0.007 times as probable.
  • prob(i vs. min): This is simply 1/prob(min vs. i). So, it means that the minimum model is x times as probable as Model i to minimize the information loss. In the Wikipedia example, the model where AIC=100 is equally probable, the minimal model is 2.7 times as probable as the model where AIC=102, and the minimal model is 148.4 times as probable as the model where AIC=110.

OK, so I think I get the mathematics somewhat, but I really don't understand what all this means in terms of practically selecting one model over another. The rule of thumb says that if the minimal model is 2.7 times as probable as another model (that is, $\Delta i$ ≤ 2), then the two models are practically equivalent. But why is 2.7 times probability in minimizing information so little as to be of no consequence? What does that even mean? Similarly, the rule of thumb says that only when you get to when the minimal model is 148.4 times as probable as another model (that is, $\Delta i$ > 10) would you say that the model under consideration can no longer be considered equivalent in any meaningful way. Isn't that an extremely huge amount of tolerance?

Even when breaking it down mathematically this way, the rule of thumb still does not make any intuitive sense to me. So, this brings me to my question:

  • Could someone please explain a simple logical justification for the commonly accepted rule of thumb for acceptable AIC differences?
  • Alternatively, could someone please provide alternative rational criteria that are better justified than this rule of thumb?
$\endgroup$
  • 1
    $\begingroup$ I don't want to make it an answer 'cause it's only my guess, that doesn't fit perfectly to the numbers: $1-\exp(-\frac{1}{2}\cdot 2)\approx0.63$, which is quite close to the $1\sigma$ level (see the 68-95-99.7 rule). Similarly, $1-\exp(-\frac{1}{2}\cdot 6)\approx0.95$ ($2\sigma$) and $1-\exp(-\frac{1}{2}\cdot 10)\approx0.993$ ($3\sigma$). Maybe the first threshold (2) was chosen to be a round number; maybe $7$ was chosen instead $6$ 'cause it's a lucky number; maybe $10$ was chosen 'cause it's a decade, so a very round number. $\endgroup$ – corey979 Jun 5 '18 at 18:57
  • $\begingroup$ @corey979, but what does $1−exp(−12⋅2)≈0.63$ actually mean, in layman's terms? Is it that 63% of models with the same real amount of information (in the sense of Akaike "Information" Criterion) would produce AIC values within that range? That would make sense, if that is what it means. (I'm not a statistician; I'm just an advanced user of statististics that tries to understand the real-world meaning behind the procedures I use.) $\endgroup$ – Tripartio Jun 7 '18 at 10:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.