7
$\begingroup$

Wikipedia's page on AIC gives a formula for the AICc, a "corrected" version of the AIC that helps to avoid overfitting when the sample size is small relative to the number of parameters in the models being considered.

Wikipedia's formula is consistent with Burnham & Anderson (2002), with the correction term $2K(K+1)/(n-K-1)$, where $K$ is the number of parameters and $n$ is the sample size.

However, in every source I have found prior to Burnham & Anderson (2002), the correction term is stated as $2(K+1)(K+2)/(n-K-2)$. For example, see Anderson et. al. (1994).

Why are the two formulas different? Was there a mistake in the original derivation that was later corrected, or did the set of assumptions change at some point? I have not been able to find any explicit mention of the difference between these two formulas.

$\endgroup$
  • 2
    $\begingroup$ One possible explanation may be that in one formula the intercept or even the variance parameter is counted and in the other formula it isn't. I don't know for sure that this is the case, but often when formulas differ by 1 in every place, it's because the variable ($K$ in this case) isn't defined exactly the same way. $\endgroup$ – Glen_b -Reinstate Monica Sep 11 '13 at 2:09
  • $\begingroup$ For the one obtained in Burnham & Anderson (2002), you should double check: Hurvich, C.M., and Tsai, C-L. (1989). Regression and time series model selection in small samples. Biometrika 76, 297–307. $\endgroup$ – Stat Sep 11 '13 at 2:15
  • $\begingroup$ @Glen_b, great idea; that's my best working theory so far. But, looking at the original sources, the definition of K does not appear to have changed before/after the formula change, which leaves me uncertain. $\endgroup$ – zkurtz Sep 11 '13 at 2:27
  • $\begingroup$ @Stat, OK, I just now looked at Hurvich and Tsai. They use the older formula, just like all the other pre-2002 sources. $\endgroup$ – zkurtz Sep 11 '13 at 2:28
  • $\begingroup$ (Anderson & Burnham 1999) is using the -1 version. As an aside, anyone know why they switched the order of the names for the book? $\endgroup$ – Avraham Sep 11 '13 at 5:35
8
$\begingroup$

I wish I understood the paper better, but if you look at Hurvich & Tsai (1989) equation 3, they are defining the AIC itself as: $$ \textrm{AIC} = n(\log\hat{\sigma}^2 + 1) + 2\left(m + 1\right) $$

Which, naïvely implies $k = m+1$ and then the Hurvich & Tsai and Anderson et al post 1999 are actually one and the same as $$ (m+1 = k) \implies \frac{2(m+1)(m+2)}{n−m-2} \equiv \frac{2(k)(k+1)}{n−k−1} $$

Edit - (Cavanaugh 1997)

See (Cavanaugh 1997) (pdf), specifically page 203, where in the derivation he is setting $k = p+1$, for it is as @Glen_b said, $k$ includes the error variance and $p$ does not.


Reference: Cavanaugh, J. E. Unifying the derivations for the Akaike and corrected Akaike information criteria Statistics & Probability Letters, 1997, 33, 201-208

$\endgroup$
  • $\begingroup$ NB $K$ in the Wikipedia/Burnham-Anderson definition includes all estimated parameters: for a general linear model, the intercept and the error variance. In Cavenaugh's paper the intercept is included in the parameter vector with length p, but the error variance is not. $\endgroup$ – Scortchi - Reinstate Monica Sep 11 '13 at 8:34
  • 1
    $\begingroup$ I mean it's included if an intercept is fitted of course. The only parameter not in the vector is the error variance. $\endgroup$ – Scortchi - Reinstate Monica Sep 11 '13 at 8:41
  • 1
    $\begingroup$ The more recent A&B work is not restricted to regression but any fitting that generates a likelihood (eg compare a weibull to a pareto MLE fit on the exact same data) so switching to a $k$ which is all the fitted parameters is sensible. $\endgroup$ – Avraham Sep 11 '13 at 8:44
  • $\begingroup$ Very true. Perhaps you could edit "intercept parameter" to "error variance" in your last sentence, as the answer is good otherwise. $\endgroup$ – Scortchi - Reinstate Monica Sep 11 '13 at 8:55
  • $\begingroup$ I have marked this as the best answer because I'm pretty sure that it must be basically correct. I'm still a little confused/annoyed at the lack of clarity in the papers: without happening to notice that the substitution m+1=k makes things "work", it seems like Cavanaugh is the only paper to alert the reader that the error variance is part of the models but not being counted in m. $\endgroup$ – zkurtz Sep 11 '13 at 15:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.