4
$\begingroup$

Thank you in advance for your time and consideration! I am a non-mathematically-inclined graduate student in communication just learning multilevel modeling.

We are running different models - some have ordered factors as the response variable and others have continuous (duration of use) or count data (frequency of use) as the response variable.

I am trying to compare the AIC for 5 different models:

  1. model.mn5 <- lmer(anxious ~ num.cm + num.pmc + (1|userid), data = df, REML = F)
  2. model.mn5.log <- lmer(log(anxious) ~ num.cm + num.pmc + (1|userid), data = df, REML = F)
  3. model.mn5.gamma.log <- glmer(anxious ~ num.cm + num.pmc + (1|userid), data = df, family = Gamma(link="log"))
  4. model.mn5.gamma.id <- glmer(anxious ~ num.cm + num.pmc + (1|userid), data = df, family = Gamma(link="identity"))
  5. model.ord5 <- clmm(anxious ~ num.cm + num.pmc + (1|userid), data = df, na.action = na.omit)

(num.cm is the group mean and num.pmc is the group-mean-centered score of the predictor)

For the models with count data (frequency of use) as the response variable, I suppose that we might also want to be able to compare Poisson and negative binomial distributions...

Despite many posts on various help forums, I understand that it's possible to compare non-nested models with different distributions as long as all terms, including constants, are retained (i.e. see Burnham & Anderson, Ch 6.7), but that different R packages or model classes might handle constants differently or use different algorithms (see point 7), thus making it difficult to directly compare AIC values. To avoid this non-comparability pitfall, it was suggested in one post to calculate your own log-likelihood (though I'm having trouble finding this post again).

Additionally, one post suggested that in order to compare normal with log-normal, you would transform the AIC for the log-normal model with the following code: AIC + 2*sum(log(anxious)). I am still unsure how to compare the lmer/normal models with the glmer/gamma models, as well as between glmer/gamma models with different link functions.

Please could you help with the following:

  • What is the best practice for comparing the AICs for these 5 models?
  • What is the R-code for manually calculating the log-likelihood and/or the AIC to retain all terms, including constants?
  • Can you compare ordinal models (clmm) with the continuous models?
  • Do you recommend any other methods and/or packages for comparing models with different distributions and/or links?

Many thanks in advance for your time and consideration! I greatly appreciate any suggestions.

Improve this question
$\endgroup$
5
  • 1
    $\begingroup$ Interesting question! One thing I try to keep in mind when formulating a statistical model is that the model will answer specific questions based on the available data, while making specific assumptions about the conditional distribution of the response variable given the predictor variables and - in your case - the random effects. If your response variable anxious is an ordinal variable, the most principled way to analyze the data on that response would be to formulate an ordinal mixed effects model. $\endgroup$
    – Isabella Ghement
    Commented Mar 5, 2020 at 16:51
  • 1
    $\begingroup$ There might be situations where you could get away with treating the ordinal response variable 'as if' it is continuous. However, if you let the data choose how to model the response variable (e.g., by comparing AIC values for competing models), that also means you are letting the data choose what research hypotheses you would like to address with your modelling. That can be a recipe for disaster, as your data-driven research hypotheses will end up reflecting the idiosyncracies of the sample at hand. $\endgroup$
    – Isabella Ghement
    Commented Mar 5, 2020 at 16:54
  • $\begingroup$ As one of my statistics professors liked to say,"we don't care about the sample at hand, except that it tells us something about the underlying population it represents". So we certainly don't want the research hypotheses we address to be driven by that particular sample. $\endgroup$
    – Isabella Ghement
    Commented Mar 5, 2020 at 16:56
  • 1
    $\begingroup$ To sum up, you would be better off in my view to decide upfront if you should treat your response variable as continuous and unbounded, continuous and bounded (e.g., proportion, percentage), discrete (e.g., count) or categorical (e.g., nominal, ordinal). Once you make this decision, what is the most principled way to proceed? Each of these types of response variables can be analyzed with different types of models, which reflect the nature of the response variable. $\endgroup$
    – Isabella Ghement
    Commented Mar 5, 2020 at 17:02
  • 1
    $\begingroup$ @IsabellaGhement - thank you for your reply! (and my apologies for the delay in responding to your help). $\endgroup$
    – user757007
    Commented Apr 22, 2020 at 19:44

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.