# Why would a smaller AIC than BIC lead to an increased chance of both overfitting and underfitting?

I am puzzled by the following statement in my lecture notes

AIC penalty smaller than BIC; increased chance of overfitting
BIC penalty bigger than AIC; increased chance of underfitting

Is there a typo?

• $$\text{AIC}-\text{BIC} = k(2-ln(n))$$ where $k$ is the number of model parameters and $n$ the number of observations. So whether $\text{AIC}>\text{BIC}$ or $\text{AIC}<\text{BIC}$ is purely dependent on the number of observations and equivalent to $n\leq 7$ and $n>7$. Commented Feb 7, 2023 at 7:26
• My hunch would be that the notes aim to say that, since AIC penalizes less heavily, it leads to larger models (or at least no smaller) than those chosen by BIC. (More specifically, AIC has a nonzero probability of choosing too large a model under certain assumptions, see e.g. stats.stackexchange.com/questions/197112/…) Hence, if the larger model chosen is too large, we have overfitting. The "underfitting" claim for BIC is a bit more tenuous to me, as BIC can be a consistent selection crit. Commented Feb 7, 2023 at 8:08
• @ChristophHanck, that makes sense. Also, if we aim for optimal prediction accuracy, then a consistent criterion underfits (there is something about it producing infinite risk, but I cannot remember the details). On the flip side of that, if we aim for recovering the true model, an efficient criterion overfits. Commented Feb 7, 2023 at 12:56

Both Akaike information criterion and Bayesian information criterion balance the number of parameters and the magnitude of the likelihood: $$AIC=2k-2\log(\hat{L}),\\ BIC=k\log (n)-2\log(\hat{L})$$ Increasing the number of parameters increases the value of likelihood - posing the risk of overfitting and reducing the magnitude of the information criteria. In the same time the first term in these criteria grows with the number of parameters. The minimum is thus reached when the linear increase in the first term is balance by the decrease due to the growth of the likelihood.

The number of data points is usually such that $$2<\log(n)$$ (indeed, the simple forms of the criteria given above are not applicable for small samples). In other words, the penalty for the number of parameters is higher for BIC than for AIC: $$k\log(n)>2k$$, and BIC has the minimum at smaller number of parameters.

Thus, smaller AIC means more risk of overfitting, whereas smaller BIC means more risk of underfitting.

(The quote in the OP is somewhat ambiguous, since it can be interpreted as referring to the bigger/smaller penalty OR as to BIC smaller/bigger than AIC.)

• Thank you. I thought it was referring to BIC smaller/bigger than AIC - which made no sense to me .But I see now the mention of penalty. Now I am confused about the difference between "AIC/BIC" and "AIC/BIC Penalty." Commented Feb 8, 2023 at 2:09
• Penalty is the first term: $2k$ or $k\log(n)$. Commented Feb 8, 2023 at 6:17

A stronger penalty will reduce the size of the fitted model.

A weaker penalty will increase increase the size of the fitted model.

AIC penalty smaller than BIC; increased chance of overfitting

BIC penalty bigger than AIC; increased chance of underfitting

The AIC penalty is smaller*, this will increase the size of the fitted model. As a consequence, the use of AIC increases the probability of overfitting in comparison to the use of the BIC.

The BIC penalty is larger*, this will decrease the size of the fitted model. As a consequence, the use of BIC increases the probability of underfitting in comparison to the use of the AIC.

• (if the number of observations is larger than 7)

No, there is no typo. The quote makes sense, or at least there is a straightforward interpretation under which it does:

• The smaller the penalty for model complexity, the more complex the selected model. The more complex the selected model, the more it will overfit. Thus smaller penalty implies greater chance of overfitting.
• The larger the penalty for model complexity, the less complex the selected model. The less complex the selected model, the more it will underfit. Thus larger penalty implies larger chance of underfitting.

There may be another interpretation, though I find it less straightforward in the context of the quote:

• If we aim for optimal prediction accuracy, we want an efficient selection criterion such as the AIC. Then a consistent selection criterion such as the BIC underfits.
• If we aim for recovering the true model, we want a consistent selection criterion such as the BIC. Then an efficient selection criterion such as the AIC overfits.

However, we aim either for optimal prediction or for recovery of the true model. Then if one criterion overfits, the other one must be optimal (cannot underfit), or if one criterion underfits, the other must be optimal (cannot overfit). Again, I do not find this interpretation as relevant as the first one, but it provides a perspective.

• I see now how the quote is ambiguous, since you and I interpreted it differently: you speak of the penalty, while I (and probably the OP) thought it was about AIC being bigger or smaller than BIC Commented Feb 7, 2023 at 16:03
• @RogerVadim, I see. I think the culprit is the slight mismatch between the title and the body of the post. First, I skimmed the title and focused on the body; I based my answer on the body alone. Then I looked at the title again and realized my answer might be not exactly what the OP is looking for. But I think it might be what the author of the quote intended! Anyway, I think all three answer are good and should help the OP solve the problem. Commented Feb 7, 2023 at 16:45