# Problem with ANOVA repeated measures: “Error() model is singular”

I need an help because I don´t know if the command for the ANOVA analysis I am performing in R is correct. Indeed using the function aov I get the following error: In aov (......) Error() model is singular

The structure of my table is the following: subject, stimulus, condition, sex, response

Example:

subject  stimulus condition sex    response
subject1    gravel  EXP1    M      59.8060
subject2    gravel  EXP1    M      49.9880
subject3    gravel  EXP1    M      73.7420
subject4    gravel  EXP1    M      45.5190
subject5    gravel  EXP1    M      51.6770
subject6    gravel  EXP1    M      42.1760
subject7    gravel  EXP1    M      56.1110
subject8    gravel  EXP1    M      54.9500
subject9    gravel  EXP1    M      62.6920
subject10   gravel  EXP1    M      50.7270
subject1    gravel  EXP2    M      70.9270
subject2    gravel  EXP2    M      61.3200
subject3    gravel  EXP2    M      70.2930
subject4    gravel  EXP2    M      49.9880
subject5    gravel  EXP2    M      69.1670
subject6    gravel  EXP2    M      62.2700
subject7    gravel  EXP2    M      70.9270
subject8    gravel  EXP2    M      63.6770
subject9    gravel  EXP2    M      72.4400
subject10   gravel  EXP2    M      58.8560
subject11   gravel  EXP1    F      46.5750
subject12   gravel  EXP1    F      58.1520
subject13   gravel  EXP1    F      57.4490
subject14   gravel  EXP1    F      59.8770
subject15   gravel  EXP1    F      55.5480
subject16   gravel  EXP1    F      46.2230
subject17   gravel  EXP1    F      63.3260
subject18   gravel  EXP1    F      60.6860
subject19   gravel  EXP1    F      59.4900
subject20   gravel  EXP1    F      52.6630
subject11   gravel  EXP2    F      55.7240
subject12   gravel  EXP2    F      66.4220
subject13   gravel  EXP2    F      65.9300
subject14   gravel  EXP2    F      61.8120
subject15   gravel  EXP2    F      62.5160
subject16   gravel  EXP2    F      65.5780
subject17   gravel  EXP2    F      59.5600
subject18   gravel  EXP2    F      63.8180
subject19   gravel  EXP2    F      61.4250
.....
.....
.....
.....


As you can notice each subject repeated the evaluation in 2 conditions (EXP1 and EXP2).

What I am interested in is to know if there are significant differences between the evaluations of the males and the females.

This is the command I used to perform the ANOVA with repeated measures:

aov1 = aov(response ~ stimulus*sex + Error(subject/(stimulus*sex)), data=scrd)
summary(aov1)


I get the following error:

> aov1 = aov(response ~ stimulus*sex + Error(subject/(stimulus*sex)), data=scrd)
Warning message:
In aov(response ~ stimulus * sex + Error(subject/(stimulus * sex)),  :
Error() model is singular
> summary(aov1)

Error: subject
Df  Sum Sq Mean Sq F value Pr(>F)
sex        1  166.71  166.72   1.273  0.274
Residuals 18 2357.29  130.96

Error: subject:stimulus
Df Sum Sq Mean Sq F value Pr(>F)
stimulus       6 7547.9 1257.98 35.9633 <2e-16 ***
stimulus:sex   6   94.2   15.70  0.4487 0.8445
Residuals    108 3777.8   34.98
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Error: Within
Df Sum Sq Mean Sq F value Pr(>F)
Residuals 420 9620.6  22.906
>


The thing is that looking at the data it is evident for me that there is a difference between male and females, because for each stimulus I always get a mean higher for the males rather than the females. Therefore the ANOVA should indicate significant differences....

Is there anyone who can suggest me where I am wrong?

Finally, I know that in R there are two libraries on linear mixed models called nlme and lme4, but I have never used it so far and I don´t know if I have to utilize it for my case. Is it the case to utilize it? If yes, could you please provide a quick R example of a command which could solve my problem?

Best regards

Dear all, I am stuck now ;-( Indeed I understood everything you suggested me but still I don´t get significance in the ANOVA results, and definitively there is an error, because results cannot be non-significant. Indeed looking at the means for each stimulus, it is possible to notice that males gave always higher evaluations than females.

To prove this I discarded for a moment the effect of the repeated measures, and I performed an ANOVA separately on both the two conditions (EXP1 and EXP2) during which the evaluations were given. What I get is significant differences between males and female, in both EXP1 and EXP2.

Now, why when I perform the ANOVA with repeated measures I don´t get the same behavior?

My design is the following: -sex is a between-subjects factor (with two levels) -stimulus is a within-subjects factor (with 3 assumed levels) -condition is a within-subjects factor (with 2 levels) -all factors are fully crossed

I tried, both the ways suggested but without achieving significance:

mDf <- aggregate(response ~ subject + sex, data=scrd, FUN=mean)
summary(aov(response ~ sex, data=mDf))     # ANOVA with just the between-effect


and

aov1 = aov(response ~ sex*stimulus*condition + Error(subject/(stimulus*condition)), data=scrd)
summary(aov1)


Instead if I perform the ANOVA on the two subtables of EXP 1 and 2 I get significant differences.

table_EXP1 <- subset(scrd, condition == "EXP1")
table_EXP2 <- subset(scrd, condition == "EXP2")

fit_table_EXP1 <- lm(response ~ stimulus*sex, data=table_EXP1)
summary(fit_table_EXP1 )
anova(fit_table_EXP1 )

fit_table_EXP2 <- lm(response ~ stimulus*sex, data=table_EXP2)
summary(fit_table_EXP2)
anova(fit_table_EXP2)


....how can this be possible?...it is a contraddiction....

HELP!

Cheers

• You don't include "condition" as a variable in your model, meaning that you are left with multiple observations per cell of the model and I'm not sure how aov deals with that. Try ezANOVA from the ez package and comment here if there are any warnings or errors. Command: ezANOVA( data=scrd , wid=.(subject) , dv=.(response) , within=.(stimulus,condition) , between=.(sex) , observed=.(sex) ) – Mike Lawrence May 21 '11 at 14:08
• This link gives a very nice explanation on how to do a repeated measures ANOVA with nlme: blog.gribblelab.org/2009/03/09/repeated-measures-anova-using-r – nico May 21 '11 at 15:53
• Maybe you could be a bit more explicit about your design: Do you have two fully-crossed within-subject IVs, stimulus and condition, meaning that each observer had all stimuli in all conditions? sex obviously is a between-subjects IV, and thus does not belong into Error(). At first glance, the formula may be response ~ stimulus * condition * sex + Error(subject/(stimulus * condition)). – caracal May 21 '11 at 16:25
• Hello everybody thanks for your comments. @Mike Lawrence: I did as you suggested using ezANOVA, but I get an error: > ezANOVA(data=scrd, wid=.(subject) , dv=.(response), within=.(stimulus,subexperiment), between=.(sex), observed=.(sex)) Error in ezANOVA(data = scrd, wid = .(subject), dv = .(response), within = .(stimulus, : unused argument(s) (wid = .(subject), observed = .(sex)) – L_T May 21 '11 at 18:45
• @nico: thanks for the link. But why should I use the nlme package? Which is the advantage rather than using aov? – L_T May 21 '11 at 18:48

Assuming your design is the following:

1. sex is a between-subjects IV (with two levels)
2. stimulus is a within-subjects IV (with 3 assumed levels)
3. condition is a within-subjects IV (with 2 levels)
4. all IVs are fully crossed

Then this is what you can do to run the full analysis, or to just test for a main effect of sex (generating some data first):

Nj        <- 10                               # number of subjects per sex
P         <- 2                                # number of levels for IV sex
Q         <- 3                                # number of levels for IV stimulus
R         <- 2                                # number of levels for IV condition
subject   <- factor(rep(1:(P*Nj), times=Q*R)) # subject id
sex       <- factor(rep(1:P, times=Q*R*Nj), labels=c("F", "M")) # IV sex
stimulus  <- factor(rep(1:Q, each=P*R*Nj))    # IV stimulus
condition <- factor(rep(rep(1:R, each=P*Nj), times=Q), labels=c("EXP1", "EXP2"))
DV_t11    <- round(rnorm(P*Nj,  8, 2), 2)     # responses for stimulus=1 and condition=1
DV_t21    <- round(rnorm(P*Nj, 13, 2), 2)     # responses for stimulus=2 and condition=1
DV_t31    <- round(rnorm(P*Nj, 13, 2), 2)
DV_t12    <- round(rnorm(P*Nj, 10, 2), 2)
DV_t22    <- round(rnorm(P*Nj, 15, 2), 2)
DV_t32    <- round(rnorm(P*Nj, 15, 2), 2)
response  <- c(DV_t11, DV_t12, DV_t21, DV_t22, DV_t31, DV_t32)       # all responses
dfL       <- data.frame(subject, sex, stimulus, condition, response) # long format


Now with the data set up, you can use aov(), but you won't get the $\hat{\epsilon}$ corrections for the within-effects.

> summary(aov(response ~ sex*stimulus*condition
+                        + Error(subject/(stimulus*condition)), data=dfL))
Error: subject
Df Sum Sq Mean Sq F value Pr(>F)
sex        1  2.803  2.8030    0.51 0.4843   # ... snip ...


You can also use the Anova() function from the car package, which gives you the $\hat{\epsilon}$ corrections. However, it requires your data to be in wide format. You have to use multivariate notation for your model formula.

> sexW  <- factor(rep(1:P, Nj), labels=c("F", "M"))     # factor sex for wide format
> dfW   <- data.frame(sexW, DV_t11, DV_t21, DV_t31, DV_t12, DV_t22, DV_t32) # wide format
> # between-model in multivariate notation
> fit   <- lm(cbind(DV_t11, DV_t21, DV_t31, DV_t12, DV_t22, DV_t32) ~ sexW, data=dfW)
> # dataframe describing the columns of the data matrix
> intra <- expand.grid(stimulus=gl(Q, 1), condition=gl(R, 1))
> library(car)                    # for Anova()
> summary(Anova(fit, idata=intra, idesign=~stimulus*condition),
+         multivariate=FALSE, univariate=TRUE)
Univariate Type II Repeated-Measures ANOVA Assuming Sphericity
SS num Df Error SS den Df         F    Pr(>F)
(Intercept)   17934.1      1   98.930     18 3263.0403 < 2.2e-16 ***
sexW              2.8      1   98.930     18    0.5100 0.4843021  # ... snip ...


Using the ez package and the command suggested by @Mike Lawrence gives the same result:

> library(ez)              # for ezANOVA()
> ezANOVA(data=dfL, wid=.(subject), dv=.(response),
+         within=.(stimulus, condition), between=.(sex), observed=.(sex))
\$ANOVA
Effect DFn DFd          F            p p<.05         ges
2       sex   1  18  0.5099891 4.843021e-01       0.004660043      # ... snip ...


Finally, if the main effect for sex is really all you're interested in, it's equivalent to just average for each person across all the conditions created by the combinations of stimulus and condition, and then run a between-subjects ANOVA for the aggregated data.

# average per subject across all repeated measures
> mDf <- aggregate(response ~ subject + sex, data=dfL, FUN=mean)
> summary(aov(response ~ sex, data=mDf))     # ANOVA with just the between-effect
Df  Sum Sq Mean Sq F value Pr(>F)
sex          1  0.4672 0.46716    0.51 0.4843
Residuals   18 16.4884 0.91602

• Hello caracal, thanks for your reply. At the moment I have a question: what do you mean exactly with IV? Sorry but my knowledge is limited....I am not a statician... – L_T May 22 '11 at 0:36
• @user4701 Sorry: IV -> "Independent Variable" or "factor" in your experiment. – caracal May 22 '11 at 8:57

Clearly sex is a between condition. You've stated below in the comments that stimulus is within subjects and condition is as well. You are only supposed to put your within conditions in the error term.

So, ...

aov(response ~ stimulus * sex * condition + Error(subject/(stimulus * condition))


Or, if as you've done it in your example it looks like maybe you don't actually want to test condition analyzed, in which case it would be...

a <- aggregate(response ~ stimulus + sex + subject, myData, mean)
aov(response ~ stimulus * sex + Error(subject/stimulus), a)

• Hi John thanks a lot. I noticed that I have some lack of terminology. What do you mean exactly when you say that stimulus is a between condition? In my experiment each subject tried 7 different stimuli, in 2 different conditions...sorry if this is a low level question....please enligthen me! ;-) And if you have a web page to suggest me where to learn more about the specific problem of this discussion please share the link ! Thanks – L_T May 22 '11 at 0:50
• Hi I studied better the terminology and I understand what between and within subjects idependent variable mean ;-) – L_T May 22 '11 at 12:33
• Editing the question so that now stimuli is also a within variable. – John May 22 '11 at 16:37
• Why is the last model not repeated measures anymore? I thought stimulus is within subjects, and you also set up the Error() term that way. – caracal May 22 '11 at 20:32
• oops... didn't finish the edit... (gone now) — BTW, they're both mixed within/between – John May 22 '11 at 21:36

I was getting this as well, because my advisor wants to see results in the ANOVA framework instead of a linear mixed effect model. R-Bloggers had a solution:

Dealing with “Error() model is singular”

Sometimes you might be unlucky enough to get this error when you try to specify your aov() object. It’s not the end of the world, it just means that you don’t have an observation for every between-subjects condition for every participant. This can happen due to a bug in your programming, a participant being noncompliant, data trimming after the fact, or a whole host of other reasons. The moral of the story, though, is that you need to find the participant that is missing data and drop him or her from this analysis for the error to go away. Or, if the idea of dropping a participant entirely rubs you the wrong way, you could look into conducting the analysis as a linear mixed model. We don’t have a tutorial for that (yet!), but keep your eyes peeled: as soon as it’s written, we’ll update this post and link you to it!

https://www.r-bloggers.com/two-way-anova-with-repeated-measures/

I agree and would suggest that you fit your model using a linear mixed model using the lme4 package. If you are tied to the ANOVA framework, look for missing values and delete them. Simply using na.omit() or which(complete.cases()) won't accomplish this, as it does it by row. And your data are in long format, so it won't delete a participant if they have any missing data.

• Another source of this error is if you include a between-subjects variable (like sex here) in the Error formula, like I did. – Rodrigo Jan 11 '19 at 15:10