To aggregate and lose resolution OR not to aggregate and suffer with correlated binary data?

Question

I have data from an experiment in which each participant provides a binary response to each presented stimulus, which is either correct (1) or incorrect (0).

There are 4 different stimulus types, and 48 stimuli from each group are presented to each participant. I'd like to compare the accuracy achieved between pairs of stimulus types (across all participants).

The obvious thing to do is to aggregate responses within each stimulus group for each participant, to arrive at an accuracy score to be used in a repeated measures ANOVA.

However, I'm bothered by the fact that this ignores the number of trials that went into computing that accuracy score - significance testing would yield the same significance whether each accuracy score was based on 48 binary responses or 400 binary responses! The other option, then, would be to do work with the raw binary data, but then the data is no longer independent, and so a chi-squared test is out of the question. I'm vaguely aware that logistic regression or generalized estimating equations (gee) could work for this, but I've only seen them used with smaller clusters of correlated data.

What is the correct thing to do here?

Answer 1

You're right that averaging over the responses and performing a repeated measures ANOVA is not the ideal thing to do. Your intuitions are good; there should be a difference between a global accuracy that's based on 48 responses and one based on 400. First, you should be using logistic regression. If you're not very familiar with that, it may help you to read the answer I wrote to this question: difference between logit and probit models. Although it was written in a different context, there's a lot of information about logistic regression, and it can help you get a sense of what it's about. Logistic regression, in its basic form, is for independent data, and your data are not independent. To deal with this, you want to either fit a logistic regression model using the generalized estimating equations, or fit a GLiMM. The choice of which to use is based on the nature of the question you want to ask. I discuss these issues in this question: difference between generalized linear models generalized linear mixed models in SPSS. For more thorough explanations of these topics, you may want to read Agresti's Introduction to Categorical Data Analysis. One thing I think you won't need to worry about is the size of the clusters though, these will work fine with your situation. HTH.