I am hitting a wall and would appreciate some help.

Here is my data (raw data, before computing the means):

   'data.frame':   8000 obs. of  11 variables:
    $ group                : Factor w/ 5 levels "asiapic","asiavid",..: 3 3 3 3 3...
$ subject              : Factor w/ 80 levels "P17001","P17002",..: 1 1 1 1 1 ...
    $ gender_part          : Factor w/ 2 levels "female","male": 2 2 2 2 2 2 2 2 ...
$ order                : Factor w/ 2 levels "first","second": 2 2 2 2 2 2 2 ...
    $ condition            : Factor w/ 2 levels "baseline","experiment": 1 1 1 ...
$ voice                 : Factor w/ 10 levels "adam","aline",..: 6 7 6 9 3 ...
    $ sex                  : Factor w/ 2 levels "f","m": 1 1 1 2 2 1 1 1 1 1 ...
$ accent               : int  3 7 4 9 4 3 4 7 8 4 ...
    $ intelligibility      : num  0.571 0.857 0.75 0.714 0.429 ...
$ item                 : Factor w/ 100 levels "90","26","50",..: 1 2 3 4 5...
    $ comprehensibility    : int  7 3 7 3 8 5 6 5 3 7 ...

My DVs are accent (values 1-9), comprehensibility (values 1-9), intelligibility (ratio 0-1).

My main IVs are group (between subjects, 5 levels) and condition (within subject, 2 levels: baseline and experiment).

I have been looking (for a very long time) for the right way to analyze these variables. What I primarily interested in is the group * condition interaction.

Were all the group same in the baseline? Were they different in the experimental condition? - when it comes to my 3 DVs - and these DVs may be slightly correlated.

Additionally, I am interested in the possible effect of sex (this is the gender of the speaker in the stimuli) and maybe participant's gender.

I have tried mixed ANOVA (for each DV separately):

model = aov(accent ~ group * condition + Error(subject/condition), data=data)

This gives me the infamous Error() model is singular, which I think it's because the lowest level will always be included here, and since I have only 2 conditions this is included by default and I can just go with Error(subject), right?

Now I'm not sure if:

(1) Is this really the right way to do it?

(2) Which function to use for the post hoc test and how to make sure that the groups were comparable in the baseline condition?

(3) Should I analyze data for male and female voices (stimuli) separately or put it somehow into the model? sex*group*condition?

I have also tried lme with lme4:

model = lmer(accent ~ sex * group * condition + (1|subject) + (1|item), data=data) 

(where item is the audio file in the stimuli)

(4) Would this be a better idea and if so how would you then deal with p-values and pair-wise comparison?

I realize this is a lot of questions but I would really appreciate some help.


1 Answer 1


A couple of notes:

  • Your depenent variables accent and comprehensebility are bounded and also seem to be discrete. Both the aov() and lmer() analysis assume normally distributed error terms. Hence, you need to see how good the normal approximation is for your data. With regard to intelligibility, this is binary, and therefore you will need to analyze it with a mixed effects logistic regression, i.e., glmer(..., family = binomial()).
  • The specification via random effects and lmer() is more flexible than the aov().
  • Fitting separate models for males and females means actually that the variance components (i.e., error variance, and variances for the random effects) are different between males and females. If you only include the interaction sex * group * condition only the fixed effects of group, condition and their interaction group:condition are different between males and females, but not the variance components. Using the latter approach, you can more easily test if you need the interaction with sex.
  • For a fitted mixed model, you can do multiple comparisons and obtain adjusted $p$-values using either the emmeans of multcomp packages.
  • $\begingroup$ - It is true that both accent and comprehensibility ratings seem to be discrete but this is the standard measure for these two dimensions (it was proved that it has a nearly 1.0 correlation to the linear rating). However, since these are bounded I am getting stripes in the residuals plot after fitting lme - which I assume I should not have. - intelligibility is a ratio words_all/words_correct, I'm sorry I was not clear about it. -I will try the package you mentioned. Thank you. - Any good method to deal with influential data points? $\endgroup$
    – Karvin
    Oct 7, 2018 at 6:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.