Can results for model selection with AIC be interpretable at the population level?

Question

AIC results for model selection are dependent on the sample size. For example if I make this model with a sample size n=100:

set.seed(123)
N <- 1000
n <- 100
r2 <- .01
y <- rnorm(n)
AIC.mod0 <- AIC(glm(y ~ 1))

> AIC.mod0
[1] 268.5385

BIC.mod0 <- BIC(glm(y ~ 1))

> BIC.mod0
[1] 273.7488

res <- replicate(N, {
  x <- y * sqrt(r2) + rnorm(n, sd=sqrt(1-r2))
  list(aic = AIC(glm(y ~ x)),
       adj.r2 = summary(lm(y ~ x))$adj.r.squared)
}, simplify=F)

> summary(sapply(res, "[[", "adj.r2"))
      Min.    1st Qu.     Median       Mean    3rd Qu.       Max. 
-0.0102041 -0.0076633 -0.0009994  0.0077916  0.0162644  0.1411315 

> summary(sapply(res, "[[", "aic"))
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
  254.3   267.9   269.6   268.7   270.3   270.5 

> summary(sapply(res, "[[", "bic"))
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
  262.1   275.7   277.4   276.5   278.1   278.4 

We can see that adjusted $R^2$ is a bit over-corrected but is nevertheless positive. AIC is higher than in the null model so it suggests that adding the variable x as a predictor does not improve the model (the same for BIC).

But if I make the same experiment with a sample size n=1000:

set.seed(123)
N <- 1000
n <- 1000
r2 <- .01
y <- rnorm(n)
AIC.mod0 <- AIC(glm(y ~ 1))

> AIC.mod0
[1] 2824.197

BIC.mod0 <- BIC(glm(y ~ 1))

> BIC.mod0
[1] 2834.013

res <- replicate(N, {
  x <- y * sqrt(r2) + rnorm(n, sd=sqrt(1-r2))
  list(aic = AIC(glm(y ~ x)),
       bic = BIC(glm(y ~ x)),
       adj.r2 = summary(lm(y ~ x))$adj.r.squared)
}, simplify=F)

> summary(sapply(res, "[[", "adj.r2"))
     Min.   1st Qu.    Median      Mean   3rd Qu.      Max. 
-0.001002  0.005103  0.008727  0.009737  0.013583  0.036300 

> summary(sapply(res, "[[", "aic"))
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
   2788    2812    2816    2815    2820    2826 

> summary(sapply(res, "[[", "bic"))
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
   2803    2826    2831    2830    2835    2841 

Adjusted $R^2$ doesn't move a lot in mean but full model AIC is now clearly lower than null model AIC, suggesting that adding the variable x improves the model. We have the same issue for BIC.

So my question could be: is it correct to reject a more complex model based on AIC knowing that a simple sample size increase could make the more complex model more efficient?

Answer 1

I am not sure if the title and the body of your question are asking about the same thing, but I will try to answer in a general way to cover both. \begin{aligned} \text{AIC}&=-2(\text{loglik}-p) \\ &=-2(n\times \text{avg(loglik)}-p) \end{aligned} where

• $n$ is the sample size used for estimation,
• $p$ is the number of the model's degrees of freedom (a measure of model's flexibility),
• $\text{loglik}$ is the log-likelihood of the sample data according to the model and
• $\text{avg(loglik)}$ is the average log-likelihood per observation.

AIC allows assessing the expected log-likelihood of the model on a new data point based on a given estimation sample. (Actually, the estimate for a single new data point is $\frac{\text{AIC}}{-2n}$.) It does so by weighting fit ($\text{loglik}$) against model flexibility ($p$). As the sample size $n$ increases, average per-observation fit ($\text{avg(loglik)}$) should improve because the estimation variance should shrink. Moreover, the weight of fit ($n$) in determining the value of AIC would grow since the log-likelihood is summed over the observations. Both of these developments point in the same direction. Consequently, AIC would prefer more complex models in larger samples.

When the estimation sample is an entire finite population, there can be no new data points to be predicted, so model choice based on AIC (motivated by achieving high likelihood on a new data point) is not directly relevant.

In an infinite population, the estimation variance would be zero and only model bias (and - when it comes to forecasting - irreducible error) would be left. The likelihood would be summed over an infinite number of points ($n=\infty$), dominating the complexity penalty ($p$). Hence, the most flexible model would be chosen by AIC. (Stricly speaking, AIC might not be defined on an infinite sample, but we can still ponder how it would behave in the limit.) As such, we would not need AIC for model selection: pure model fit (the likelihood) would be enough.

Given the considerations above, I think AIC-based model selection is hard to interpret on a population level.