2
$\begingroup$

I performed ordered logistic regression using the polr() command in R. I did this with the same independent and dependent variable (the X and Y are the same for every model), but the exact data in each model is slightly different (different sample sizes, since I wanted to run a regression for different age cohorts and the sample sizes in these cohorts are not the same).

Question is: can I use the AIC or Residual deviance (output of the model in the summary) to compare the goodness-of-fit of the models? (< AIC / < residual deviance = better model)? Can I conclude that?

$\endgroup$
3
  • 1
    $\begingroup$ Several threads here on Cross Validated indicate that AIC requires the dependent variance to be exactly the same in all models. Your question might even be a duplicate of some older thread. You can find some of them here, but perhaps you can narrow down the search further by using relevant keywords and their combinations. $\endgroup$ Commented Jun 19, 2019 at 13:00
  • $\begingroup$ Is there some reason why you didn't include age or age cohort as a covariate in a combined model, potentially with interactions between age and the other predictors? That would seem to be a more powerful approach than a set of separate models for each age cohort. $\endgroup$
    – EdM
    Commented Jun 19, 2019 at 14:01
  • $\begingroup$ Yes, I considered that. However, there are repeated measures: people appear in more than one agecohort (in other words: they are tested multiple times over the years, makeing them end up in the datasets of > 1 agecohort), and my question is: in which age cohort is this X the best predictor of Y? I could not figure out how to do that within one model, therefore I chose this approach. Do you know by any chance? $\endgroup$
    – Pannie
    Commented Jun 19, 2019 at 14:03

1 Answer 1

2
$\begingroup$

No. You can't. Your AIC will vary according to the likelihood of your model, and that likelihood is determined by how "believable" the estimated parameters are given the data you inputed.

You can easily see that if you fit the same type of model with different training data, whether is results in the same parameters or not, AIC will be different in each case.

Any comparison you make between models fit with different data will tell you more about how "well behaived" each dataset is than about the quality of the models

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.