14
$\begingroup$

As motivated by the recent change of the default model selection statistic in the R's forecast package from AIC to AICc, I am curious whether the latter is indeed applicable wherever the former is. I have a series of questions with this respect and here is the first one.

I know that to replace AIC with AICc everywhere is what the well-known book in (1) by Burnham and Anderson (non-statisticians), as summarized here, recommends. The book is sometimes uncritically referred to by younger statisticians, see e.g. comments to this blog post by Rob Hyndman, but the statistician Brian Ripley advised in a radically different way:

“Burnham and Anderson (2002) is a book I would recommend people NOT read until 
they have read the primary literature. I see no evidence that the authors have 
actually read Akaike’s papers." [quoted from [AIC MYTHS AND MISUNDERSTANDINGS][4] by
Burnham-Anderson]

It does follow from what Ripley writes on the AIC and related theory that the warning should be taken seriously. I have both a good collection of Akaike's own papers and the Burnham-Anderson book. I will eventually have my own opinion on the quality of the book, but it will also help to know what the community of statisticians, both young and old, think on that. In particular, are there professors of statistics (or other good students of statistics) who explicitly recommended the book as a useful summary of knowledge on using AIC for model selection?

Reference:

(1) Burnham, K. P. & Anderson, D. R. Model selection and multimodel inference: a practical information-theoretic approach Springer, 2002

PS. In reply to the recent "answer" stating that "Dr.Burnham is a Ph.D. statistician" I'd like to add this clarification. Yes, by himself he is a statistician, a Fellow of the ASA and the recipient of numerous professional awards, including Distinguished Achievement Medal from the ASA. But who says he is not? All I have said above is that as a pair of authors they are not statisticians and the book reflects this fact.

$\endgroup$
  • 2
    $\begingroup$ The paper "AIC Myths and Misunderstandings" can be found here. I hadn't seen it (though I'd seen Ripley's comment before). $\endgroup$ – Glen_b -Reinstate Monica Feb 15 '14 at 5:15
  • 3
    $\begingroup$ The question provides links but I think it would be helpful for readers if the question itself gave some hint of what might be bad about the content of Burnham and Anderson's book. (If what they say is accurate, clear, helpful, etc., then it doesn't matter whether they actually read Akaike's papers.) Also, it seems to me that AIC and related methods are still controversial; if so, then any book presenting them will have detractors. And a suggestion that everyone must read the original papers before reading a book that aspires to provide an introduction to a topic seems questionable. $\endgroup$ – Mars Feb 15 '14 at 6:16
  • 7
    $\begingroup$ I've read Akaike's two main papers, and Schwarz's one on the BIC, and Burnham and Anderson's book (it's on my shelf right now), and also (as you see), Ripley's brief critique. I would like very much to see Ripley explain the criticism in detail (without the aspersions about who may have read what) -- as things stand, there's really nothing substantive for Burnham and Anderson to reply to. If there's something to it (and there may be for all I know), it deserves more than a few lines on an R-help mailing list. $\endgroup$ – Glen_b -Reinstate Monica Feb 15 '14 at 6:46
  • 2
    $\begingroup$ @Gleb_b Ripley's views can be deduced from his 1996 book on pattern recognition, to which he referred to in that R list post, see e.g. chapter 2. The mathematical details and comments on other aspects of Akaike's work (e.g. what does A in AIC stands for) make me think he understands this area well and had read more than just a few of Akaike's papers. $\endgroup$ – Hibernating Feb 15 '14 at 10:25
  • 3
    $\begingroup$ Putting in a different two cents: How often do AIC and AICc give different advice? In my experience they suggest the same models. $\endgroup$ – Peter Flom - Reinstate Monica Feb 15 '14 at 12:36
3
$\begingroup$

The OP appears to be seeking a high-quality survey of high-quality statisticians to help assess whether one particular book is of high quality particularly with regards to the AIC versus AICc debate. This site is not particularly geared towards systematic surveys. Instead I'll try to address the underlying question directly.

The AIC and AICc both score models according to a heuristic tradeoff between model fit (in terms of the likelihood) and overfit (in terms of the number of parameters). In this tradeoff, the AICc gives slightly greater penalty on the number of parameters. Thus, the AICc always recommends in favor of models that are of complexity less-than-or-equal to the complexity of the best AIC model. In this sense the relationship between the two is very simple, despite the horribly complicated arguments underlying their derivations.

The AIC and AICc are only two out of a large field of candidate information criteria, with the BIC and DIC being perhaps the leading alternatives. The BIC is far more conservative (penalizing large numbers of model parameters) than either of the AIC or AICc in most cases. The question of which criterion is the best is truly problem specific. One could legitimately prefer an extremely conservative criterion in cases where robust out-of-sample prediction is needed.

FWIW, I found the conservatism level of the AICc to be typically preferable over the AIC in extensive simulation studies on the prediction error in capture-recapture models.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.