Ramifications of small + unbalanced group sizes, small number of groups for fixed & random effects models?

Question

I have a variable (call it 'group') that I would like to treat as a random effect in a logistic regression. However, the number of groups is small (9 groups, larger than the recommended absolute minimum of 5 but not by much), and the sample size in each group is small and unbalanced (one group <10 observations, four groups 30-40 observations, two groups 70-90 observations, two groups 100+ observations). (I am primarily interested in the effects of the other predictors in the regression, not 'group').

I notice that if I treat 'group' as a fixed effect rather than as a random effect, it has only a minor impact on the results - standard errors of the predictors I care about are slightly smaller, and their coefficients are slightly closer to zero, when 'group' is treated as a random rather than a fixed effect, but effectively the same results.

Which predictors come out significant are also the same regardless of whether I treat 'group' as a fixed or random effect, but they change if I exclude 'group' from the model altogether.

So my question is: In a situation where initial considerations suggest that a variable should be treated as a random effect, but there are small + unbalanced group sizes as in my example (for a smallish but acceptable number of groups), and the researcher is interested in the betas of the significant predictors: is it advisable to 'back off' to a model that treats the group variable as a fixed effect, or perhaps even to a model that does not include the group variable at all?

If not, what caveats should be included with the interpretation of the random effects model (i.e., would it be accurate to state that standard errors are likely to be underestimated, and may be closer to those of a model that does not include the group variable)?

Answer 1

Mixed models are good at coping with unbalanced designs. This is one of their advantages compared to other approaches such as ANOVA-type models. So I would not worry about this.

You mention small group sizes, but the numbers you mention, are not, in my opinion, small. You will see in my answer to your other question that I was simulating data with maximum group sizes of 6 or 7 even in the second "large" simulation. Again. mixed models are very robust to small group sizes. In another answer, I showed that the mimumum cluster size is 1, under some mild assumptions.

Small numbers of groups is a bit more of a problem because the random effects are treated as multinormally distributed so if you had, for example, 2 groups, it is obviously folly to treat them as random as there could be no hope of obtaining a sensible estimate for a normally distributed variable from a sample size of 2. There is some consensus around 6 being perhaps the minimum. Of course all situations are different and your mileage may vary. With 9 groups I think you are fine, but I would certainly explore the fixed effects model, and compare the results of interest. If there is meaningful differences then you would have quite an interesting problem. In that case I would probably report the results of both models.

Lastly, try not to be too concerned with "significant" predictors. Effect sizes are far more important. Statistical significance depends a great deal on things like sample size, and is arbitrary (you might mistakenly discard an important variable because it's p value was 0.050001 while retain one that should not have been there in the first place because it's p -value was 0.04999). Covariates should be included in your model based on sound clinical / expert knowledge, ideally informed by a principled approach such as a DAG in order to avoid over-adjustment and erroneously conditioning on mediators or colliders.