I have two independent groups A and B. Each participant is given 5 questions to solve and his/her answer is evaluated as correct or incorrect.

However, participants can skip questions, so the number of attempted questions for each participant varies.

So, I end up with the data below:

     Group   # of correct answers  # of attempted answers   Correct Ratio
p1    A            3                       5                    0.6
p2    A            2                       4                    0.5
p3    B            1                       1                    1.0
p4    B            1                       3                    0.33

I want to test the whether the answers' correctness is different between group A and B. Which statistical test should I use?

Currently I simply count the total # of attempted answers and # of correct answers for all people from group A and B, say 90/100 vs 75/90, and use proportion test (in R as prop.test). But I don't know whether it is appropriate?


1 Answer 1


It depends on what you want to do with skipped answers. You can a) Assume they are questions the participant didn't know, and count them as errors (or, if the test is timed, this could be a penalty for slow work). b) Assume they would have been guessed at randomly (if the test is multiple choice). c) Assume nothing about them and ignore them. In a) the total number correct would be the score; in b) The total number correct + 1/p*m (where p is the number of choices and m is number of skipped answers) would be the score. In c) the score could be the proportion, as you have measured it.

In any of these, if you have only two groups and no other variables to add, you could do a t-test between groups, if the number of questions is pretty high. Technically, a) and b) would be counts, so t-test is not exactly appropriate, but, if there were enough questions, it would be OK.

Another alternative is regression. For a) and b) you could use a count model (Poisson or negative binomial regression). In c) you could use logistic, on the proportion of correct answers.

  • $\begingroup$ Thanks. I handle skipped answers as in (c). Still, I wonder whether I can test the equal of proportion of correct answers for group A and B, using a z-test (prop.test in R)? How is it different from t-test? $\endgroup$
    – Ida
    Feb 9, 2014 at 14:43
  • $\begingroup$ prop.test gives a chi-square test. See this thread $\endgroup$
    – Peter Flom
    Feb 9, 2014 at 15:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.