# How to fit linear mixed model with nested fixed factors with lme4

I'm using linear mixed model with the lme4 package in R to analyze EEG amplitude in two different subject groups (control & experiment).

All subjects were instructed to view images of two different categories (car vs human). While category 1 solely comprises of whole cars, category 2 can then be divided into three sub-categories (head, trunk, extremities).

My dataset looks like this:

Amplitude Group Category Sub-category Subject
13.231 control car car 1
12.123 control car car 1
18.423 control human trunk 1
14.132 control human extr 1
13.231 exper car car 2
12.123 exper car car 2
18.423 exper human trunk 2
14.132 exper human extr 2
13.412 exper car car 3
12.534 exper car car 3
15.423 exper human trunk 3
17.122 exper human extr 3
... ... ... ... ...

Now I am interested in the effect of group, category, as well as sub-category on EEG amplitude. I am also interested in the interaction between group and category, and group and sub-category.

Hence, I would like to define category and sub-category as fixed effects. My current model looks like this:

AMP.model = lmer(amplitude ~ group * (category + subcategory) + (1|patient), data=AMP)

But then i receive the following warning:

fixed-effect model matrix is rank deficient so dropping 2 columns / coefficients

I believe this is due to the assumption in linear mixed models that fixed effects should not be correlated / nested. But how else would I get the information that I am interested in?

Please note that it is not an option to remove the categories and only check for sub-categories, as my main hypothesis is based on the effects of categories on EEG amplitude.

• Seems to me your model is over-specified, and that's why you get a singular fit. I suggest using amplitude ~ group * category + group:category:subcategory + (1|patient) I don't think you need to set up special model contrasts, and the emmeans package will detect the nested structure and provide for sensible means and comparisons. Commented Dec 29, 2020 at 1:29
• Actually I think the emmeans package will deal with the singular model you have, detect the nesting structure, etc. Commented Dec 29, 2020 at 1:33

There is no way around the constraint at the stage of fitting the model; however, you should still be able to answer the questions you want. Since this problem is not specific to mixed models (i.e. it applies just as well to a simple linear model with the same structure), I'll illustrate with your data above in an lm() fit:

Set up a custom contrast matrix: I find it's easier to define these by defining the inverse contrast matrix first.

contr_inv <- matrix(c(
1/2, 1/6, 1/6, 1/6,
1,-1/3,-1/3,-1/3,
0, 2/3,-1/3,-1/3,
0, -1/3,2/3,-1/3),
byrow=TRUE,nrow=4,
unique(dd$Sub.category)) )  The first contrast is the overall intercept, treating the groups as units (1/2 * mean of group 1 + 1/2 * mean of the subcategories in group 2; the second contrast is the comparison between groups. Contrasts 3 and 4 are the comparisons between the first two subcategories of group 2 and the mean of the other two groups. Set up a factor and assign the contrasts: dd$Sub.category <- factor(dd$Sub.category, levels=unique(dd$Sub.category))

summary(m1 <- lm(Amplitude ~ Sub.category, data=dd))

This doesn't completely solve your problem, as you might want to (1) test whether there is an overall difference among subcategories within group 1 (which would translate to testing the joint null hypothesis that the two subcategory parameters (head_vs_mean and trunk_vs_mean) were both 0, which is normally done by testing whether the sum of squares of the parameters is zero) and (2) test the difference of the third subcategory (extr) from the mean of group 2. You can use multcomp::glht() or possibly emmeans to construct these differences, but I haven't gotten to that yet ...