I have data from an experiment in which each participant provides a binary response to each presented stimulus, which is either correct (1) or incorrect (0).

There are 4 different stimulus types, and 48 stimuli from each group are presented to each participant. I'd like to compare the accuracy achieved between pairs of stimulus types (across all participants).

The obvious thing to do is to aggregate responses within each stimulus group for each participant, to arrive at an accuracy score to be used in a repeated measures ANOVA.

However, I'm bothered by the fact that this ignores the number of trials that went into computing that accuracy score - significance testing would yield the same significance whether each accuracy score was based on 48 binary responses or 400 binary responses! The other option, then, would be to do work with the raw binary data, but then the data is no longer independent, and so a chi-squared test is out of the question. I'm vaguely aware that logistic regression or generalized estimating equations (gee) could work for this, but I've only seen them used with smaller clusters of correlated data.

What is the correct thing to do here?

  • $\begingroup$ Why not construct a linesr model where the variance is inversely proportional to the group sample size? $\endgroup$ – Michael Chernick Aug 10 '12 at 1:09

You're right that averaging over the responses and performing a repeated measures ANOVA is not the ideal thing to do. Your intuitions are good; there should be a difference between a global accuracy that's based on 48 responses and one based on 400. First, you should be using logistic regression. If you're not very familiar with that, it may help you to read the answer I wrote to this question: difference between logit and probit models. Although it was written in a different context, there's a lot of information about logistic regression, and it can help you get a sense of what it's about. Logistic regression, in its basic form, is for independent data, and your data is not independent. To deal with this, you want to either fit a logistic regression model using the generalized estimating equations, or fit a GLiMM. The choice of which to use is based on the nature of the question you want to ask. I discuss these issues in this question: difference between generalized linear models generalized linear mixed models in SPSS. For more thorough explanations of these topics, you may want to read Agresti's Introduction to Categorical Data Analysis. One thing I think you won't need to worry about is the size of the clusters though, these will work fine with your situation. HTH.

  • $\begingroup$ Thanks gung! I'll have some GEE vs. GLMM decision-making to do - now the next question is how to properly use (and interpret the output from) SPSS to accomplish this... $\endgroup$ – Will Aug 10 '12 at 21:58
  • $\begingroup$ No problem, @Will, if you have any questions about GEE vs GLiMM's or any confusions, you can comment there & ping me as well. GL $\endgroup$ – gung Aug 10 '12 at 22:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.