I have the following data:

There are three conditions, A, B and C (presented in random order). Each condition has two difficulties (i.e., six total). Each respondent is scored as 1 (if correct), or 0 (if incorrect). Each respondent did each condition once. With this we get the following:

Condition          Correct
A Easy             19/29
A Hard             16/29
B Easy             10/29
B Hard              0/29
C Easy             13/29
C Hard              6/29

I want to statistically prove that respondents are more correct in one condition compared with another. I stumbled upon the McNemar's test, but since B Hard is a constant, that will not work. Any other suggestions?

  • $\begingroup$ So each respondent provides 6 responses, is that right? How many respondents do you have in total? Did no one ever get B Hard correct? $\endgroup$ Aug 11, 2015 at 20:43
  • $\begingroup$ Yes, nobody had B Hard correct, there were 29 respondents, so in B Hard 29 were incorrect. $\endgroup$
    – Swifting
    Aug 11, 2015 at 20:47
  • $\begingroup$ Are you interested in testing the interaction Condition X Difficulty? $\endgroup$ Aug 11, 2015 at 21:02
  • $\begingroup$ Are you trying to determine if the Hard conditions are statistically more difficult than the Easy ones? Are you trying to compare A to B to C? $\endgroup$
    – MikeP
    Aug 11, 2015 at 21:02
  • $\begingroup$ Both, comparing Easy with Hard, and A with B. $\endgroup$
    – Swifting
    Aug 11, 2015 at 21:04

2 Answers 2


Because your respondents are providing multiple scores, you need to account for the non-independence in your data. The primary two ways to do that are to fit a generalized linear mixed effects model (GLMM), or use the generalized estimating equations (GEE). To understand how these differ, and to decide which you might prefer, it may help you to read my answer here: Difference between generalized linear models & generalized linear mixed models in SPSS. My guess is that you would prefer the GLMM.

The catch is that GLMMs are finicky and sometimes just won't fit. The 0 corrects in B Hard is especially likely to cause problems. If you don't include an interaction term, you might be OK, though. If you did try to fit an interaction term you will have separation (also known as the Hauck-Donner effect), and the GLMM fit is unlikely to converge. Even if you don't include the interaction, and the fit does converge, you should probably fit nested models and use likelihood ratio tests instead of Wald tests.


Have you considered Fisher's Exact Test? https://en.wikipedia.org/wiki/Fisher%27s_exact_test

Using it I get p-values for Hard vs Easy of 0.1545 for A, 0.0004 for B, and 0.0340 for C. Therefore, I think that you can probably reject the null hypothesis (no difference between easy and hard) for B and C, but not for A.

For A, I setup the table as follows:

enter image description here

choose(29,19)*choose(29,16)/choose(58,35) = 0.1545

Similar analysis (sum of easy and hard for each) shows all the types (A, B, and C) are different from one another (A and B p value is 1.4e-6, A and C p value is 0.0018, and B and C p value is 0.027).

  • $\begingroup$ Hmm but can you do a Fisher's Exaxt Test on paired data? $\endgroup$
    – Swifting
    Aug 11, 2015 at 23:15
  • $\begingroup$ I believe so, the examples I've seen generally are looking at whether there is a difference in one set of paired outcomes based on another set of paired characteristics. I also just noticed your comment from an hour ago about which test is most effective. I would say which ever is answered correctly closest to 50% of the time gives you the most information. $\endgroup$
    – MikeP
    Aug 11, 2015 at 23:25
  • $\begingroup$ -1, Fisher's exact test does not take the non-independence of the data into account. This suggestion is inappropriate for the OP's situation. $\endgroup$ Aug 12, 2015 at 3:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.