I have a big dataset, where I have thousands of controls and then subjects with different conditions. In total, I have around 26 classes (conditions), including controls. The data is very unbalanced. For example, I have thousands of controls, then a few hundred in one class, and other classes have less than 10 subjects. I'm interested in looking at associations between brain measures and risk for disease, associated with these conditions. I have a variable that reflects that risk, with zero being the controls. However, I'm concerned about the unbalanced problem, since I have so many controls and a different number of subjects in each condition. I am afraid the effect will be driven by differences between controls and subjects with conditions, instead of truly reflecting an association between the different risk associated with conditions and brain measures.

At first, I thought about subsampling my dataset, but in a previous question, I posted here, I was advised to use linear mixed models, where I could use my whole dataset. I have been playing with this, but I'm not totally sure on how to apply this to my problem. I'm using the package lm4 in R and the function lmer().

I tried this:

glmm1<- lmer(brain_measure ~ risk + Age + Sex  + ICV + (1|condition),data=dataset_all,REML = F, na.action = na.exclude)

This model provided interesting results, where some were similar to when using lm() and others were different. However, I also receive a warning about singularity issues in some brain measures (boundary (singular) fit: see ?isSingular). After reading some threads I'm not sure if this should be a concern. Also, condition and risk are obviously related, since different conditions have different risks, and, for example, there are a lot of subjects with risk=0 given that most of the people are controls. I don't know if this is a problem.

I would like your advice on this. If this is the best approach, and if I'm applying this right.

Thank you!

  • $\begingroup$ What happens when your run a variance components model that has just your outcome and your random intercept? Do you get the singularity problem? $\endgroup$
    – Erik Ruzek
    Apr 26, 2021 at 1:06
  • $\begingroup$ Hi @ErikRuzek. Do you mean without the + Age + Sex + ICV covariates? Yes, I still have the singularity problem. I only have this problem for some brain regions though. $\endgroup$
    – Blueberry
    Apr 26, 2021 at 9:09
  • $\begingroup$ Can you post the results of glmm1<- lmer(brain_measure ~ 1 + (1|condition), data=dataset_all, REML = F, na.action = na.exclude)? I don't understand how you are discerning that the problem is in certain brain regions. $\endgroup$
    – Erik Ruzek
    Apr 26, 2021 at 18:56
  • $\begingroup$ @ErikRuzek I think perhaps there are several different models? Each with a different brain measure as DV. Is that right, @Blueberry? In any case, Erik is getting at the fact that singularity warnings usually happen when a variance component is close to zero (on the boundary of the parameter space since we don't want negative variances). Also, I'm not sure that conditions are exchangeable if control is meaningful. That is, it's not really a random effect is it? $\endgroup$
    – Rick Hass
    Apr 26, 2021 at 20:55
  • $\begingroup$ @RickHass yes, that is right. I'm running different models for each region, and I would correct for multiple comparisons at the end. I think you're right about the exchangeability of conditions, but then I'm not sure how to model the variance between conditions and take into account the difference in sample sizes in the model. $\endgroup$
    – Blueberry
    Apr 26, 2021 at 22:36

1 Answer 1


This is a tough one and I'm going to answer based on an important issue for mixed effects / multilevel models. The issue is exchangeability, which is basically a model specification issue. Every text on multilevel models discusses exchangeability and the choice of model specification in different ways. Snijders and Bosker (2012) do a nice job of discussing the related issues on p. 45-46. Under point 1, they assert that if the groups (in your case "conditions") are considered unique categories, and you want to draw conclusions about each of these categories (including control), then a regular ANCOVA will do it. One reason is that in modeling the "condition" as random, you're assuming that the group effects are independent and identically distributed. That is, the groups are exchangeable, much like persons are exchangeable in an ordinary, single level regression.

The reason that this is tough is that obviously you have unbalanced data and the number of conditions you have is large such that perhaps they are not distinct. Multilevel / mixed effects models are helpful in this situation. However, there seems to be a systematic difference between control and non-control cases, which conceptually makes it difficult to model the "condition" or lack thereof as a random effect.

It also helps to put the model in hierarchical form to think about this. Following your choice of condition as random, with only random intercepts, you have the following:

Level 1: person-level

$$ y_{ij} = \beta_{0j} + \beta_{1j}*risk + \beta_{2j}*Age + \beta_{3j}*Sex + \beta_{4j}*ICV + r_{ij} $$

Level 2: condition level

$$ \beta_{0j} = \gamma_{00} + u_{0j} $$

The $u_{0j}$ are the (random) group effects, and are considered random variables (i.i.d.). But is the $u_{0j}$ for the control group really drawn from the same distribution as the $u_{0j}$ for all other groups? There isn't an easy answer to this question. Either way, the singular fit warning will sometimes happen when estimating the variance of the $u_{0j}$ if the variance is close to zero (on the boundary of the parameter space). You will still get parameter estimates but they can be invalid because maximum likelihood estimation assumes that the parameters are not on the boundary of the space. Hope this helps.

  • $\begingroup$ Thank you for your answer, this indeed made more sense to me. If it helps to understand the sample, the conditions I'm referring to are genetic variants. So, I have a sample with thousands of controls, with no variants, and then I have subjects with different variants and each variant I called "condition". Some variants are more frequent than others, making this dataset very unbalanced. I want to know if the risk that each variant represents to certain disease is associated with changes in the brain, and therefore I'm looking at the association between risk and brain measure. $\endgroup$
    – Blueberry
    Apr 26, 2021 at 22:09
  • $\begingroup$ I was advised to do subsampling by randomly selecting 5 subjects in each "condition" and doing linear regression, repeating this process around 1000 times (or more). But I'm concerned that by doing this I will miss more subtle effects and I'm not taking advantage of the whole unique sample I have in hands. On the other hand, by using the whole sample using mixed models, I'm afraid that most of my effect could be driven by differences between controls (with no genetic variants) and carriers of these variants. I'm not sure what the best way to proceed is here. $\endgroup$
    – Blueberry
    Apr 26, 2021 at 22:14
  • $\begingroup$ You could test for that by introducing a fixed effect at level-2 that is binary: variants v no variants. However with the singular fit issue, there may not be much between group variability. Good luck! $\endgroup$
    – Rick Hass
    Apr 27, 2021 at 0:51
  • $\begingroup$ (+1) great answer, Rick !!! $\endgroup$ May 27, 2021 at 22:19
  • $\begingroup$ Thanks! I appreciate the feedback $\endgroup$
    – Rick Hass
    May 28, 2021 at 23:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.