Is it true that if models A and B are such that the covariates of B are a subset of A, the AIC of model A is always smaller than that of model B? Why?
To add on F. Tusell's answer: the AIC (but also other model selection criteria, such as the BIC or the HQC) contains two terms:
- A likelihood term, which measures the ability of the model to explain the observed data;
- A regularization term, which penalizes the complexity of the model (i.e. its number of free parameters).
The point of the latter term is precisely to avoid overfitting and to allow that the selected model will generalize well to unobserved data, i.e. to avoid that the more complicated model will be necessarily selected. This can be interpreted as a form of Occam's razor: if two models explain the data equally well, the simpler one should be favored.