# How to analyse binary responses for various factors, including interactions: chi square, mixed models, logistic regression, or ANOVA on percentages?

I run an experiment where subject had to recognize an emotion from various musical stimuli (which were composed with a certain emotional intent). There were 4 levels of emotional_intent, subjects results of a forced choice task were either 1 for correct, and 0 for wrong. There were 2 groups of participants, musicians and non-musicians. So the dependent variable is binary, the within subject factor is emotional_intent with 4 levels, and the between subject factor is musical_expertise. My goal is to understand wether there are differences in the recognition 1) between groups overall, 2) between emotions, 3) between groups within each emotional_intent level.

What is the best way to analyze the data? I considered these options:

1. Chi square: I compare the number of correct and wrong answers for the groups, and for the emotions, and for the interaction of these two factors. That seems simple, but what are the disadvantages here?

2. Use a mixed effect model and use post hoc test on the factors and interaction term Here I got a problem because being the answers either 0 or 1, the data are clearly not normally distributed. So the use of the anova on the fitted model resulting from the mixed effect would break the normality assumption. Am I correct, or can I still use the mixed effect model?

This is what I would use in R:

fit <- lmer(correct ~ emotional_intent* musical_expertise + (1|subject), data=scrd)
anova(fit)

# Post hoc of the interaction term
emmeans(fit, pairwise~emotional_intent*musical_expertise, adjust = "tukey")

1. Use a binomial logistic regression followed by an ANOVA: Here I could do an ANOVA on a model fitted using a binomial logistic regression, and then I can compute the post hoc tests on that model:
model <- glm(correct ~emotional_intent * musical_expertise,family=binomial(link='logit'),data=scrd)
anova(model, test="Chisq")

# Post hoc of the interaction term
emmeans(model, pairwise~emotional_intent*musical_expertise, adjust = "tukey")


1. I calculate the percentage of correct answers for each subject, then I compute a two-way ANOVA with repeated measures and the usual post-hoc tests. Here data are normally distributed, and assumptions are not violated. However, using the method at step 2, I get that there is actually statistical significance in the interaction term and in the related pairwise comparisons of the posthoc test.

Which method is more correct to use and why? Are there other methods?

• How many statistics courses have you taken? What books have you read? Dec 16, 2020 at 12:47
• I think what @Frank is probing for is some clues about what statistical facts and concepts we may assume in our answers and what we would have to explain. Although it's nice you have published papers, that doesn't really help us in that regard.
– whuber
Dec 16, 2020 at 13:53
• What you are describing as a post hoc of the interaction term is actually simply comparisons of the four cell predictions, wherein both marginal effects and interaction effects are in play. If you need all 6 of those comparisons, fine. But you might consider doing just the four simple (non-diagonal) comparisons. Dec 17, 2020 at 16:47
• I agree a logistic model is a better choice than the lmer() model. But there is also the possiblity of using glmer(), wherein you can account for subject variations. There is also the possibility of a GEE model. Somebody besides me could comment on the relative merits of these methods, but I do think it's important to account for the subject effects. Dec 17, 2020 at 18:34
• Subjects are part of the design, and it is reasonable to expect that they will not all respond the same. Dec 17, 2020 at 23:27

The study on the various techniques mentioned in the original question above led me to conclude that the right approach is a binomial logistic regression with subject as a random effects. The R code I used is the following

library(lme4)
model <- glmer(correct ~ emotional_intent * musical_expertise + (1|subject),family=binomial(link='logit'),data=scrd)
summary(model)

car::Anova(model)

library(emmeans)
# Post hoc of the interaction term
emmeans(model, pairwise~ emotional_intent * musical_expertise, adjust = "tukey")
$$$$
`