2
$\begingroup$

I'm running logistic mixed-effects models for a project using glmer(), but ran into a few problems with model fit.

In this model, there are 2 fixed effects:

  1. Factor A, a continuous variable (but only with 3 values, -25, 0, and 25)
  2. Factor B, a categorical variable with 4 levels

The only random effect structure in the model is (1|subject).

I collected data from 2 different samples for this project. For one of the samples that used this model structure I ended up getting a singularity warning, and the random effect/standard deviation is 0. The model ran just fine for the sample, but the random effect was relatively small - when I removed the random effect and ran a regular logistic regression using this second sample, and compared the regular logistic regression vs mixed effects logistic model using anova.Mermod(), the addition of the random effect didn't seem to contribute significantly to the glmer model with the random effect. Removing the random effect structure for models that previously ran into singularity issues also seemed to fix the problem (and it didn't give me weird coefficient estimates like in previous cases).

So this all seems good, except that removing the random effect of subject would also violate assumptions of independence (even though subject doesn't seem to contribute a significant amount of variance), and is it okay to violate this assumption in this particular scenario? If not, what would be some alternative solutions to this problem (the only thing I can think of is to run a mixed-ANOVA, treating Factor A as 3 levels of a discrete factor, and then following up with linear contrasts to look at the effect of Factor A at each level of Factor B). Any advice would be much appreciated!


EDIT: I ended up running GEEs to get around this issue, since the goal was not to model subject as a random factor, but rather to account for the within-subject variance.

$\endgroup$

2 Answers 2

1
$\begingroup$

I collected data from 2 different samples for this project.

Is there a reason to model the two samples completely separately? You could model them together and assume the random-effects variances were the same in each sample, e.g.

glmer(response ~ A*B*sample + (1|subject), ...)

? This will fit separate coefficients for A, and B, and their interactions for each sample. You would need to decide how to parameterize the model: if you want completely separate coefficients for each sample, you'd use ~(A*B):sample+sample-1, e.g.

dd <- expand.grid(A=c(-25,0,25),B=factor(letters[1:4]),sample=factor(1:2))
colnames(model.matrix(~(A*B):sample+sample-1, data=dd))

If you wanted instead to parameterize by mean effect and deviation between mean and sample 1, you'd use sum-to-zero contrasts: ~A*B*sample with contrasts=list(sample=contr.sum)

For one of the samples that used this model structure I ended up getting a singularity warning, and the random effect/standard deviation is 0. The model ran just fine for the [other] sample, but the random effect was relatively small - when I removed the random effect and ran a regular logistic regression using this second sample, and compared the regular logistic regression vs mixed effects logistic model using anova.Mermod(), the addition of the random effect didn't seem to contribute significantly to the glmer model with the random effect. Removing the random effect structure for models that previously ran into singularity issues also seemed to fix the problem (and it didn't give me weird coefficient estimates like in previous cases).

In general a model with a singular fit (random effects variance = 0) for a single random effect grouping will give exactly the same results as the model with the random effect removed. If the random-effects variance is small (in magnitude, not necessarily significant/non-significant!) then the difference will be small but not zero.

So this all seems good, except that removing the random effect of subject would also violate assumptions of independence (even though subject doesn't seem to contribute a significant amount of variance), and is it okay to violate this assumption in this particular scenario?

Opinions differ widely on this (see e.g. the GLMM FAQ section on singular fits. I personally prefer not to drop random-effects terms, just because they're small or non-significant, especially when they're part of the experimental design.

If not, what would be some alternative solutions to this problem

I would probably just report the results from the singular fit. A singular fit is not necessarily wrong; it just suggests that the random effect may be small and/or poorly constrained by the data. I would prefer to analyze the two samples together (see above), but I don't see that it's required to drop the random effect from the second sample for consistency.

See the link above for lots of other possibilities (e.g. using a Bayesian prior to constrain the random-effects variance away from zero, as in the blme package).

$\endgroup$
3
  • $\begingroup$ Hi Ben, first of all, thanks for taking the time to respond to my question. My response is a little long so I'm going to have to separate it into a few different comments. The reason for why the samples are analyzed separately is because they are from two different experiments. The choice to drop the random effect of subject isn't to be consistent per se; there were other problems when I later break down the interaction (i.e., analyze the effect of Factor A at each level of Factor B). 1/2 $\endgroup$
    – monajhzhu
    May 14, 2020 at 19:00
  • $\begingroup$ For some models, including the random effect of subject actually produces an estimate that does not make sense in the context of the study (e.g., 0.07, which is roughly what it should be vs 30). When the random effect is removed, the estimates for a regular glm is much more reasonable. To me, this suggest that there are other issues with including the fixed effect. I've also looked into GEEs (which I know far less about); while the estimates seem to match the ones in the glm and glmer models, the std error is several magnitudes different than the ones in a glm/glmer model. 2/2 $\endgroup$
    – monajhzhu
    May 14, 2020 at 19:04
  • $\begingroup$ Hi again Ben. I ended up running GEEs instead, since the goal was merely to account for within-subject variance rather than to model it. The issue I had earlier was due to not factorizing the subject variable (ha). Thanks again for your help on this! $\endgroup$
    – monajhzhu
    May 14, 2020 at 23:49
0
$\begingroup$

After receiving some helpful feedback from folks from methods discussion group, I was informed that generalized estimation equations would potentially get around this issue (and it did after I ran the model). For this particular project, subject was largely a nuisance variable, and the initial intention for using mixed-models was to account for the within-subject variance/non-independence of observation. Since we didn't actually need to model the random effect of subject (in that we didn't need to obtain estimation/information about this variable), GEEs seemed better-fitted for the job anyways. Running GEEs also solved some of the other issues I was getting (i.e., getting estimates that were off by several magnitudes and would not have made sense in the context of the current experiment).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.