0
$\begingroup$

I have a handful of logistic regression models and I would like to report AIC. Should I report it on training or test data? I have quite a big dataset and a maximum of 10 predictors in any of the models.

$\endgroup$
5
  • $\begingroup$ Are you talking about the Akaike Information Criterion? I would report the AIC on the validation data. I would break down my dataset to train test validation. You would need to report it on data having the same outcome otherwise AIC wouldn't make much sense $\endgroup$ Aug 10, 2022 at 10:17
  • $\begingroup$ I only have train/test data. Train data have the same outcome in all models. Right now, I calculate in train data. Should I run the model in the whole dataset and report that AIC? $\endgroup$
    – lola
    Aug 10, 2022 at 10:54
  • $\begingroup$ I would use the test data then. however it also depends on whether you want to report how you trained your model. If it is important to be detailed on the decisions you took to train the models, AIC is good for comparisons $\endgroup$ Aug 10, 2022 at 11:13
  • 2
    $\begingroup$ AIC is penalized for complexity. If your dataset is so huge (e.g., N > 20000) so that you have the luxury of splitting the data, you don't absolutely have to penalize for complexity in the test data when the model was frozen after fitting on the (independent) training data. The gold standard would be the out-of-sample deviance (-2 log likelihood) to report on the very large test dataset. $\endgroup$ Aug 10, 2022 at 11:18
  • $\begingroup$ By "The gold standard would be the out-of-sample deviance (-2 log likelihood) to report on the very large test dataset." you mean the complete dataset? $\endgroup$
    – lola
    Aug 10, 2022 at 11:24

1 Answer 1

1
$\begingroup$

Model fit can be evaluated by (estimated) expected likelihood $L$ or (estimated) expected log-likelihood $\ln(L)$ on new data. Multiplying $\ln(L)$ by a constant $-2$ does not change the essence but turns out to be convenient in certain applications. Now, $\text{AIC}=-2\ln(L)+2p$ incorporates a penalty $2p$ needed for de-biasing the estimated expected log-likelihood $\ln(L)$ when evaluating the model on the same data that was used for fitting the model. Thus if you evaluate your model on the training data, it makes sense to use AIC.

On the other hand, when the model is evaluated on test data (not the same as the training data), there is no bias to $-2\ln(L)$. Therefore, it does not make sense to penalize it by $2p$, so using AIC does not make sense; you can use $-2\ln(L)$ directly.

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.