3
$\begingroup$

I've read two interpretations of Akaike's Information Criterion (AIC) that seem to be in conflict, and I was hoping that someone could help me understand how to reconcile them.

Interpretation 1: AIC's approximation to the model's KL divergence is only accurate up to an unknown constant, so AIC is useless for estimating predictive error in a more absolute way. This is what Burnham and Anderson say about it.

Interpretation 2: AIC is an approximation to the out-of-sample prediction error, which is on the same absolute scale as cross-validation. This is the approach that Gelman and Vehtari take.

So who's right? Can they both be correct?

$\endgroup$
2
$\begingroup$

Looking at that Gelman and Vehtari reference, they seem to define a cross-validation criterion which resembles a likelihood - so that isn't interpretable as an absolute measure, in the same way AIC isn't. In other words, the "out-of-sample prediction error" measure they use is in itself not an "absolute" measure of fit quality, in the sense that it can't be used to measure fit quality per se.

Does that sound right?

I'd be surprised if either of those author pairings got the technical details around AIC wrong. My problems with B&A's approach tend to be more on the implementation...and the rhetoric...

$\endgroup$
  • $\begingroup$ Thanks, maybe this is the crux of the issue: Why isn't likelihood interpretable as an absolute measure (assuming it's been scaled in a sensible way)? If R-squared is absolute, and it can be calculated from the likelihood, then where does the unknown constant come from? $\endgroup$ – David J. Harris Oct 4 '14 at 1:05
  • $\begingroup$ Or is R-squared only relative in this sense, too? It's hard to think of a definition of "relative" where that would make much sense... $\endgroup$ – David J. Harris Oct 4 '14 at 1:05
  • $\begingroup$ Likelihood is a function of sample size (and other things, like support of your data) as well as model fit. That's why it can't be used as a generic measure of model fit, but two models fitted to exactly the same data set can be compared using likelihoods. $\endgroup$ – Mark Brewer Oct 4 '14 at 8:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.