How to fit linear mixed model with nested fixed factors with lme4

Question

I'm using linear mixed model with the lme4 package in R to analyze EEG amplitude in two different subject groups (control & experiment).

All subjects were instructed to view images of two different categories (car vs human). While category 1 solely comprises of whole cars, category 2 can then be divided into three sub-categories (head, trunk, extremities).

My dataset looks like this:

Amplitude	Group	Category	Sub-category	Subject
13.231	control	car	car	1
12.123	control	car	car	1
11.876	control	human	head	1
18.423	control	human	trunk	1
14.132	control	human	extr	1
13.231	exper	car	car	2
12.123	exper	car	car	2
11.876	exper	human	head	2
18.423	exper	human	trunk	2
14.132	exper	human	extr	2
13.412	exper	car	car	3
12.534	exper	car	car	3
19.233	exper	human	head	3
15.423	exper	human	trunk	3
17.122	exper	human	extr	3
...	...	...	...	...

Now I am interested in the effect of group, category, as well as sub-category on EEG amplitude. I am also interested in the interaction between group and category, and group and sub-category.

Hence, I would like to define category and sub-category as fixed effects. My current model looks like this:

AMP.model = lmer(amplitude ~ group * (category + subcategory) + (1|patient), data=AMP)

But then i receive the following warning:

fixed-effect model matrix is rank deficient so dropping 2 columns / coefficients

I believe this is due to the assumption in linear mixed models that fixed effects should not be correlated / nested. But how else would I get the information that I am interested in?

Please note that it is not an option to remove the categories and only check for sub-categories, as my main hypothesis is based on the effects of categories on EEG amplitude.

Answer 1

There is no way around the constraint at the stage of fitting the model; however, you should still be able to answer the questions you want. Since this problem is not specific to mixed models (i.e. it applies just as well to a simple linear model with the same structure), I'll illustrate with your data above in an lm() fit:

Set up a custom contrast matrix: I find it's easier to define these by defining the inverse contrast matrix first.

contr_inv <- matrix(c(
    1/2, 1/6, 1/6, 1/6,
    1,-1/3,-1/3,-1/3,
    0, 2/3,-1/3,-1/3,
    0, -1/3,2/3,-1/3),
    byrow=TRUE,nrow=4,
    dimnames=list(c("intercept","group","head_vs_mean","trunk_vs_mean"),
                  unique(dd$Sub.category))
    )

The first contrast is the overall intercept, treating the groups as units (1/2 * mean of group 1 + 1/2 * mean of the subcategories in group 2; the second contrast is the comparison between groups. Contrasts 3 and 4 are the comparisons between the first two subcategories of group 2 and the mean of the other two groups.

Set up a factor and assign the contrasts:

dd$Sub.category <- factor(dd$Sub.category,
                          levels=unique(dd$Sub.category))
contrasts(dd$Sub.category) <- solve(contr_inv)[,-1]

summary(m1 <- lm(Amplitude ~ Sub.category, data=dd))
                          Estimate Std. Error t value Pr(>|t|)    
## (Intercept)                14.2012     0.5606  25.334 4.18e-11 ***
## Sub.categorygroup          -2.8510     1.1211  -2.543   0.0273 *  
## Sub.categoryhead_vs_mean   -1.2983     1.0028  -1.295   0.2219    
## Sub.categorytrunk_vs_mean   1.7963     1.0028   1.791   0.1008    

This doesn't completely solve your problem, as you might want to (1) test whether there is an overall difference among subcategories within group 1 (which would translate to testing the joint null hypothesis that the two subcategory parameters (head_vs_mean and trunk_vs_mean) were both 0, which is normally done by testing whether the sum of squares of the parameters is zero) and (2) test the difference of the third subcategory (extr) from the mean of group 2. You can use multcomp::glht() or possibly emmeans to construct these differences, but I haven't gotten to that yet ...