0
$\begingroup$

If I have two samples (n=300) from the same population and I'm fitting a GLMM (Generalised Linear Mixed-effects Model) with a similar response and explanatory variables but with a completely different set of participants,

M.1 <- glmer(y1 ~ x1 + x2 + (1 | id), data = Sample1 , family = binomial)

M.2 <- glmer(y1 ~ x1 + x2 + (1 | id), data = Sample2 , family = binomial)

can I compare the AIC values of these two models?

I understand that AIC = -2ln(Likelihood|data) + 2K and I'm not sure whether the likelihood|data1 and likelihhod|data2 are comparable.

However, I'm confused by the statement in, "AIC MYTHS AND MISUNDERSTANDINGS" by Anderson and Burnham, point 2 which states " Information-theoretic approaches can only be applied when there is one data set" is a myth. https://sites.warnercnr.colostate.edu/anderson/wp-content/uploads/sites/26/2016/11/AIC-Myths-and-Misunderstandings.pdf

I'm confused and any clarification is greatly appreciated. Thanks!!

$\endgroup$
0
$\begingroup$

The note is correct that the likelihoods of two different models need to be computed over the same data set. Thus, they say "one data set".

The authors of the note above, also wrote the book "Model Selection and Multimodel Inference: A Practical Information-Theoretic Approach", where in the title of subsection 2.11.1 they explicitly say that "AIC cannot be used to compare the models of different data sets". This is consistent with the note.

Since you have two different data sets obtained from two different groups of participants and use one model -- then there are no models to compare using AIC or other methods.

However, you can compare the responses, for example, how much the mean responses (of two data sets) differ and whether the difference is statistically, or substantively significant, and what conclusions you may make about the difference between the two groups of participants, etc.

$\endgroup$
  • $\begingroup$ Thanks a lot for the answer and the great reference. The note I have mentioned is also from the authors of the popular book, "Model Selection and Multimodel Inference, A Practical Information-Theoretic Approach". So wondering whether I misunderstood what they are referring to. $\endgroup$ – Jessie Apr 4 at 21:09
  • $\begingroup$ To answer your question further, could you briefly describe the data, collection/sampling method and the goal of your modeling exercise? $\endgroup$ – dnqxt Apr 5 at 0:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.