1
$\begingroup$

Problem/Question in short:

I have estimated 5 generalized linear mixed models and subsequently compared their levels of relative fit according to AIC. These models are based on a very large dataset of 3,231,544 observations related to 184,113 participants. In order to present an intuitive measure of differences in relative fit, I have computed delta AIC and weight of evidence. According to guidelines suggested by Anderson (2007), a delta AIC value of 4 constitutes a strong difference and >8 a very strong difference between a given model and the best fitting. However, I am unsure if these thresholds are still meaningful with such a large dataset. The absolute AIC values are very high, so differences between models in hundreds/thousands of raw AIC scores might not make too much difference?

My question: can I still rely upon these conventional thresholds for AIC, and if not, are there better ways to compare my models?

More background information and model fit information:

Study: I am studying longitudinal gambling behavior of 184,113 participants. The data is based on complete tracking of electronic gambling behavior within a gambling operator. Gambling behavior data is aggregated on a monthly level, a total of 70 months. I have an ID variable separating participants, a time variable (months), as well as numerous gambling behavior variables such as active days played for a given month and gambling loss limits reached.

Analytical approach: Generalized linear mixed modelling was done on count variables active gambling days and gambling loss limits with the glmmTMB package (Brooks et al., 2017). Both models have a negative binomial 2 probability distribution function. For active gambling days, a zero-truncated model was used as months without active gambling days were treated as missing. For gambling loss limits, a zero-inflated model was used. I have used a model comparison approach to identify the best approximate model of longitudinal differences in indicators of high-risk gambling associated with age and biological sex. For each outcome variable, five models were run:

  1. an unconditional means model with a fixed and random (individual specific) intercept,
  2. an unconditional growth model with a fixed time effect and random (individual specific) time effect,
  3. a conditional age model updated from the previous model with age category as a fixed effect,
  4. a conditional age and sex model updated with fixed main and interaction effects for age and sex, and
  5. a conditional age, sex and time model, updated with a three-way interaction between age, sex and time.

Models were evaluated against the Akaike’s Information Criterion (AIC).

Results from regarding model fit comparisons:
enter image description here

My interpretation based on conventional thresholds mentioned above: The results indicate that for gambling loss limits, model 4 has the best fit with delta AIC and weight of evidence indicating strong effect sizes between model 4 and other candidate models. For active days gambling, the results indicate more uncertainty with model 4 and model 5 performing similarly as evident by equal AIC scores, delta AIC and weight of evidence. One could choose model 4 in favour of the more parsimonious model, i.e. fewer predictors.

References:

Brooks, M. E., Kristensen, K., Van Benthem, K. J., Magnusson, A., Berg, C. W., Nielsen, A., . . . Bolker, B. M. (2017). glmmTMB balances speed and flexibility among packages for zero-inflated generalized linear mixed modeling. The R journal, 9(2), 378-400.

Anderson, D. R. (2007). Model based inference in the life sciences: a primer on evidence: Springer Science & Business Media.

This book has especially guided my understanding of model fit so far: Long, J. D. (2012). Longitudinal data analysis for the behavioral sciences using R: Sage.

$\endgroup$
1
  • $\begingroup$ An interesting aspect of the topic is that the justification behind AIC and similar metrics is asymptotic. Thus, when your data is really large, AIC should have its nice properties. $\endgroup$ – Richard Hardy Feb 19 at 14:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.