3
$\begingroup$

I have two linear mixed-effects models using the same sample, the same random effects and the same response variable. They differ in a single fixed effect. One of the models seems to give better estimates (no convergence warnings) while using the BOBYQA optimizer. Is it then a legitimate comparison to compare the AIC of the two models that have estimates calculated in slightly different ways?

$\endgroup$

1 Answer 1

6
$\begingroup$

Short answer: You can always "compare" them, but the devil is in the details --- i.e., how you plan to describe the meaning of the outcome of the comparison.


Longer answer: Whether or not it is valid to "compare" the AIC across two models really depends on the nature of the comparison you are making, and what you intend to infer from this comparison. The Akaike Information Criterion (AIC) is really just a penalised maximised log-likelihood value, where the statistic is calculated so that the maximum log-likelihood is negated. Thus, a lower AIC represents a "better fit" in the sense of having a higher maximised log-likelihood subject to a penalty on the parameters. In principle, this statistic can be compared between models with different distributional families, since these all involve a likelihood function for the data, and some number of fitted parameters. While a comparison can certainly be made, the devil is in the details ---i.e., how you plan to describe the meaning of the outcome of the comparison.

In classical statistics, for a comparison across nested models, one can invoke distributional results for the difference in the AIC under the null hypothesis of the restricted model, and this allows formal testing of the hypothesis that the restricted model is "better" than the larger model. (These results are very similar to the standard chi-squared tests for nested models.) It sounds like your analysis involves nested models (i.e., one model includes an additional term, while the other excludes it). This would mean that you can apply standard tests for goodness of fit.

More broadly, for a comparison involving non-nested models, all you can really say is that one AIC is lower than the other, and thus has a better "goodness of fit" as measured by the penalised maximised log-likelihood. In this latter case, the differences in the AIC for two non-nested models does not necessarily have any useful distributional result, so the comparison is somewhat vague, insofar as you don't have a known null distribution under the hypothesis that one of these model forms is the true model.

In Bayesian statistics, comparisons of AIC across models can be considered under Bayesian analysis, with the penalties on the parameters interpreted as a log-prior (see e.g., Forster and Sober 2011). This analysis means that you are essentially holding a prior belief that models with more parameters are less probably a priori, and this leads to an implicit prior penalty on those models. Interpretation of the AIC under this approach is discussed in the linked paper, which I would recommend reading.

$\endgroup$
2
  • $\begingroup$ Thank you for your answer, and I agree with what you have said. This is not quite my question. I mean that I have two different models, both fit with ML, with different fixed effects representing two different hypotheses that I would like to compare using AIC for model selection. So far, this should normally be fine. However one of the models fits without convergence warnings only using a BOBYQA optimizer. So can I be comparing the two different models using AIC if they are fit with two different algorithms? Thanks again $\endgroup$
    – Roasty247
    Jun 10, 2019 at 13:18
  • $\begingroup$ Sorry - I misread the details of your question. I have amended my answer to deal with the situation you describe. $\endgroup$
    – Ben
    Jun 10, 2019 at 14:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.