I have one fixed effect (group) and one nested random effect (nested). There are 20 levels of group, and all but one of them has 4 levels of nested. There is significant variance in nested that I feel should be included in the ANOVA model, but because a single level of group has only a single corresponding nested level, this seems technically inappropriate.

Moreover, when I run a mixed model anova while including nested (either with lme4 in R or proc MIXED in SAS), there is always some message about "Convergence criteria met but final hessian is not positive definite" (SAS warning). I found an informative SAS paper that mentioned that this error is often caused by mis-specified models (see https://support.sas.com/resources/papers/proceedings12/332-2012.pdf). However, I also saw another question on this site that seemed pertinent and whose answer implied that it was a valid approach (Is it inappropriate to do a One-way independent ANOVA when one of the levels has only one participant?).

From what I could interpret from a different question on this site, if a random effect has only one level then the mixed model's optimizer will really only be estimating the random effect based on rounding error (Mixed model runs well in R whereas a random effect has only one level). In my opinion this seems fine in my case since 1) I just need to account for the effects of nested on group, 2) if the estimate for nested in the single troublesome group level is based on rounding error then it probably won't dramatically change the estimate for that level's fixed effect, and 3) the benefit to accounting for nested within all other group levels seems worth the tradeoff.

Note that I cannot exclude the troublesome group level from the analysis.

QUESTION: Should I fit a mixed model that includes nested, and if so, with how much caution should I interpret the results?

  • $\begingroup$ Are there only 4 levels of nested? or are there 4*19+1 levels of nested? Are you sure you want to specify a fixed effect with 20 levels? Usually there are more levels of the variable to be used as a random effect than the one to be used as a fixed effect. $\endgroup$
    – Noah
    Commented Apr 18, 2020 at 21:56
  • $\begingroup$ There are 4*19+1 levels of nested. Yes, I have to specify a fixed effect with 20 levels given the experimental design. $\endgroup$ Commented Apr 18, 2020 at 22:01

1 Answer 1


This sounds like a case of partial clustering, which frequently occurs in clinical psychology treatment evaluation studies, in which individuals exposed to the treatment are clustered within therapists, but individuals in the control condition are not (i.e., they are all in one group). There has been some literature on this matter and well-specified guidelines on how to proceed when using linear mixed models. The general advice is to allow the group with only one level of clustering to be the reference group, and then to allow random slopes on the coefficients for the other levels but not a random intercept. This is critical to maintaining accurate type I error rates if the clusters are thought to be random (e.g., therapists randomly selected from a population of therapists). Some papers discussing this issue include Sterba (2017) and Baldwin et al. (2011).

In your case, this would involve setting the group with one level to be the reference level and then placing random slopes on the coefficients for the 19 other levels. You should ensure that the random effects covariance matrix is specified to be compound symmetric (which it is with mixed ANOVA). This is more easily done with PROC MIXED.

  • $\begingroup$ If the experimental design suggests a different level of group as the reference, is there another strategy? This is the case in my scenario, where it was a planned experiment and it wouldn't make sense to report the results with reference to a non-control level. $\endgroup$ Commented Apr 18, 2020 at 23:09
  • $\begingroup$ Also, is there a more general strategy than setting the troublesome group level as the reference, since these experiments are performed regularly and it could easily be the case that there are multiple troublesome group levels (i.e. which only have a single corresponding nested level)? $\endgroup$ Commented Apr 18, 2020 at 23:13
  • $\begingroup$ I think you have to allow the 1-level group to be the reference category because that would be the only way to prevent it from having a random component by omitting a random slope. If the reference group has nesting, then you have to allow a random intercept, which forces all group means to vary. You should just fit your model and perform any comparisons you want using planned contrasts afterward. $\endgroup$
    – Noah
    Commented Apr 18, 2020 at 23:19
  • $\begingroup$ With multiple groups with only one level, you still need the reference group to be one of those with one only level, but the others that only have one only level can be entered as normal, just with no random slopes on them. You get to choose where you put the random slopes so you can omit the groups that shouldn't have it. $\endgroup$
    – Noah
    Commented Apr 18, 2020 at 23:21
  • $\begingroup$ I've never heard of specifying which levels of a factor should have random slopes, either in R or SAS. How would I go about doing that? $\endgroup$ Commented Apr 18, 2020 at 23:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.