1
$\begingroup$

AIC is simply penalised log-loss, and log-loss depends directly on the dataset size. To create a model from data, missing data need to be excluded first. Assuming missing data are spread across variables, more complex models will tend to be built on smaller datasets, because more observations will need to be excluded. Consequently, more complex models will automatically have lower AIC scores, although that measure doesn't properly reflect their performance on the population.

To compensate for that effect, I thought of normalising the AIC score, simply dividing it by the size of the dataset actually used in building the model. Is this approach legitimate?

If not, why not? I'd appreciate an intuitive, practical example where the normalisation goes wrong.

Is BIC, which uses the dataset size in the penalty term, immune from this effect? I have my doubts, because the dataset size enters only logarithmically into the BIC score, while the log-loss rises linearly...

P.S. There is a similar question on CV, but it concerns with normalising the log-loss only. I think in my approach I avoid the mistake described there.

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ Though there is randomness involved, you should be fine on average. (This presumes missingness is not associated with the model fit for the particular data point. Otherwise you get a systematic bias.) BIC is not immune; you would treat it just like AIC. $\endgroup$
    – Richard Hardy
    Commented Mar 23, 2023 at 8:42

1 Answer 1

Reset to default
2
$\begingroup$

If I understand you correctly, you are trying to compare AIC values for models fitted on different subsets of your sample. Since the subsets have different size, the scales of AICs differ. To combat that, you obtain average AICs (i.e. AIC per observation) and compare them.

Though there is randomness involved, you should be fine on average. This presumes missingness of data is not associated with the model fit for the particular data point. Otherwise you may get a systematic bias. E.g. if the data tends to be missing for points that would fit terribly, excluding them will yield an overly optimistic average likelihood and thus average AIC.

The problem that you outline for AIC applies also for BIC just as much.

Improve this answer
$\endgroup$
2
  • $\begingroup$ It appears to me that my approach might be wrong. I've read "on the internet" (no reputable source at hand) that, in order for the models to be comparable using the AIC, they all have to be built on same data. Is that true, can you comment on that? Or is this "same data" requirement (if true!) only due to possible different model sizes, so my approach compensates for it? $\endgroup$
    – Igor F.
    Commented Mar 24, 2023 at 10:05
  • $\begingroup$ @IgorF., there are several posts on Cross Validated that say exactly that. This is required for a strictly sound comparison of raw AICs. However, AIC per observation is just an estimate of something (twice the negative expected likelihood of a new observation). Estimates can be obtained from different samples. As long as these samples faithfully represent the same population (in your case, this presumes missingness of data is not associated with the model fit for the particular data point), we can compare the estimates. $\endgroup$
    – Richard Hardy
    Commented Mar 24, 2023 at 10:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.