First, let us generate some data:

df <- data.frame(names = factor(1:10))
df$probs <- c(0, 0, runif(8, 0, 1))
df$response = lapply(df$probs, function(i){
  rbinom(5000, 1, i)  

dt <- data.table(df)

dt <- dt[, list(response = unlist(response)), by = c('names', 'probs')]

dt <- 
  dt %>%
  group_by(names) %>%
  mutate(probs_observed = mean(response))

dt$probs_logit <- boot::logit(dt$probs_observed)
dt$probs_logit[dt$probs_logit < -100000] <- -100000

such that dt looks like:

> tail(dt)
Source: local data frame [6 x 5]

   names     probs response probs_logit probs_observed
  (fctr)     (dbl)    (int)       (dbl)          (dbl)
1     10 0.9446753        1    2.931879         0.9494
2     10 0.9446753        1    2.931879         0.9494
3     10 0.9446753        1    2.931879         0.9494
4     10 0.9446753        1    2.931879         0.9494
5     10 0.9446753        1    2.931879         0.9494
6     10 0.9446753        1    2.931879         0.9494

A variant of this dataset was previously used here.

Consider two regression models:

lm1 <- glm(data = dt, formula = response ~ names, family = 'binomial')
lm2 <- glm(data = dt, formula = response ~ probs_logit, family = 'binomial')

I.e. use the category as the sole covariate in the first model, and the logit of the observed response rate in the category as the covariate in the second model.

These models logically should produce the same predicted probabilities, thus achieve the same log likelihood. Indeed:

> logLik(lm2)
'log Lik.' -17987 (df=2)
> logLik(lm1)
'log Lik.' -17987 (df=10)

Where this gets interesting is that the AIC of lm1 is 35994, and the AIC of lm2 is 35978. Considering the difference in degrees of freedom is 8, this makes sense since $2 * 8 = 16$ which equals the difference in AIC.

Yet, to me, these models have the same information content. How can the AIC be different?

Why does AIC penalise lm1, where the only difference is the number of categories?

  • 3
    $\begingroup$ It would help you get more responses if you ask a question in language-neutral fashion. Not everyone knows R syntax. $\endgroup$
    – Aksakal
    Feb 26 '16 at 12:37

If you start with two pre-specified models & fit them to a response using maximum-likelihood estimates of their free parameters, then the difference in AIC is an estimate of the difference in expected information loss between the two. What you've done is to define the predictor of your second model in terms of the response—that's quite a different procedure, and there were certainly not only two degrees of freedom to it. As it invariably produces two equivalent models, the expected information loss between them is known to be zero.

The usual calculations for confidence intervals, p-values, &c., will also be invalid when the response is used to define the predictors: in general frequentist inference relies on the notion of the performance of a procedure over a long series of repetitions; & estimates based on your supposed carrying out of a single, well-defined procedure are of little relevance when in fact you've done something else entirely.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.