1
$\begingroup$

I am running two longitudinal models for two different populations I'd like to compare. N1=4,000 individuals (translated into about 20,000 rows; 18 variables) and N2=400,000 individuals (~4 million rows; 15 variables). Since I am interested in the individual-level inference of my outcome, I have been using GLIMMIX with random intercept in SAS. This works very well for population 1, but not for population 2 (error message: data is too large). I also have tried to work in R, and my model still did not converge. A GEE model, however, runs smoothly for population 2.

It's been a very frustrating experience to work on this large dataset, so my questions are: (1) Do you know if the results of GEE and GLMM would, perhaps, be similar/comparable in such huge dataset? (2) Anyone else has worked with such large data using mixed effects model as well? Anything worked? Thanks.

$\endgroup$
1
  • 1
    $\begingroup$ GEE and GLMM answer different questions. I don't use SAS - is it running out of memory ? I have run GLMMs on huge datasets - if it is too large to run locally then I fire up a large EC2 instance on AWS and run it there (using R) $\endgroup$ Commented Apr 9, 2019 at 9:41

1 Answer 1

2
$\begingroup$

A couple of points:

  • Both the GEEs and GLMMs are used for cluster/grouped data, and account via different ways for the correlations in the measurements within each cluster/group.
  • The GEEs is a semiparametric approach that does not make any assumptions for the distribution of your data; you only specify the first two moments (in classic GEEs, there are extensions in which you specify higher moments). On the contrary, with GLMMs you have a full specification of the distribution.
  • Because GEEs are semi-parametric you can only use Wald tests, but no likelihood ratio tests nor information criteria based on the likelihood (e.g., AIC or BIC). Moreover, because its semi-parametric nature, the standard GEE is only valid under the missing completely at random assumption, which seldom holds in practice for all the missing data you may have. The GLMMs, on the other hand, are valid under both missing completely at random and missing at random.
  • The GEEs provide you with marginal / population-averaged coefficients and inferences, whereas GLMMs with subject-specific ones. For more info on this point check here.
  • In your specific setting, you may want to consider working with a smaller random sample from the second cohort. Often you do not need all this high-volume data to extract useful conclusions.
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.