Why would a smaller AIC than BIC lead to an increased chance of both overfitting and underfitting?

Question

I am puzzled by the following statement in my lecture notes

AIC penalty smaller than BIC; increased chance of overfitting
BIC penalty bigger than AIC; increased chance of underfitting

Is there a typo?

Answer 1

Both Akaike information criterion and Bayesian information criterion balance the number of parameters and the magnitude of the likelihood: $$ AIC=2k-2\log(\hat{L}),\\ BIC=k\log (n)-2\log(\hat{L}) $$ Increasing the number of parameters increases the value of likelihood - posing the risk of overfitting and reducing the magnitude of the information criteria. In the same time the first term in these criteria grows with the number of parameters. The minimum is thus reached when the linear increase in the first term is balance by the decrease due to the growth of the likelihood.

The number of data points is usually such that $2<\log(n)$ (indeed, the simple forms of the criteria given above are not applicable for small samples). In other words, the penalty for the number of parameters is higher for BIC than for AIC: $k\log(n)>2k$, and BIC has the minimum at smaller number of parameters.

Thus, smaller AIC means more risk of overfitting, whereas smaller BIC means more risk of underfitting.

(The quote in the OP is somewhat ambiguous, since it can be interpreted as referring to the bigger/smaller penalty OR as to BIC smaller/bigger than AIC.)

Answer 2

A stronger penalty will reduce the size of the fitted model.

A weaker penalty will increase increase the size of the fitted model.

AIC penalty smaller than BIC; increased chance of overfitting

BIC penalty bigger than AIC; increased chance of underfitting

The AIC penalty is smaller*, this will increase the size of the fitted model. As a consequence, the use of AIC increases the probability of overfitting in comparison to the use of the BIC.

The BIC penalty is larger*, this will decrease the size of the fitted model. As a consequence, the use of BIC increases the probability of underfitting in comparison to the use of the AIC.

---

• (if the number of observations is larger than 7)

Answer 3

No, there is no typo. The quote makes sense, or at least there is a straightforward interpretation under which it does:

• The smaller the penalty for model complexity, the more complex the selected model. The more complex the selected model, the more it will overfit. Thus smaller penalty implies greater chance of overfitting.
• The larger the penalty for model complexity, the less complex the selected model. The less complex the selected model, the more it will underfit. Thus larger penalty implies larger chance of underfitting.

There may be another interpretation, though I find it less straightforward in the context of the quote:

• If we aim for optimal prediction accuracy, we want an efficient selection criterion such as the AIC. Then a consistent selection criterion such as the BIC underfits.
• If we aim for recovering the true model, we want a consistent selection criterion such as the BIC. Then an efficient selection criterion such as the AIC overfits.

However, we aim either for optimal prediction or for recovery of the true model. Then if one criterion overfits, the other one must be optimal (cannot underfit), or if one criterion underfits, the other must be optimal (cannot overfit). Again, I do not find this interpretation as relevant as the first one, but it provides a perspective.