Is it true that if models A and B are such that the covariates of B are a subset of A, the AIC of model A is always smaller than that of model B? Why?
$\begingroup$
$\endgroup$
1
-
2$\begingroup$ And to state the obvious: AIC does not need nested models. $\endgroup$– usεr11852Commented Sep 27, 2022 at 18:18
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
No, it is not true. The penalty term $2p$ may give you a worse AIC for the model with more parameters.
$\begingroup$
$\endgroup$
To add on F. Tusell's answer: the AIC (but also other model selection criteria, such as the BIC or the HQC) contains two terms:
- A likelihood term, which measures the ability of the model to explain the observed data;
- A regularization term, which penalizes the complexity of the model (i.e. its number of free parameters).
The point of the latter term is precisely to avoid overfitting and to allow that the selected model will generalize well to unobserved data, i.e. to avoid that the more complicated model will be necessarily selected. This can be interpreted as a form of Occam's razor: if two models explain the data equally well, the simpler one should be favored.
