My data is many thousands of responses to ~2000 multiple-choice (2 to 4 options) questions. Most of the questions have not been answered by most people. ie. a typical question will have <5% response rate.

I'm interested in finding out which questions have a high predictive power for which other questions. eg. People who answer option A to Q243 are five times as likely than average to answer option C to Q64.

I have constructed a (4x4) answer matrix for every question-pair combination that records (of those who answered both questions under consideration) how many people answered each possible combination of answers. This looks something like this:

                Qb
           | 0  1  2  3
         --|-------------
          0| 3  12 5  2
       Qa 1| 11 2  25 10
          2| 16 5  10 7
          3| 10 2  10 17

So, the Probability that a random person (who answered both questions) selected Qa option 1, given that they selected Qb option 2 is: 25 / (5 + 25 + 10 + 10) = 0.5

In order to explore the predictive power of Qb option 2 for Qa option 1. I would want to compare this number, 0.5 to the probability that a random person would choose option Qa option 1 given no further information. ie. if 50% of the population in general, chose option 2 for Qa anyway then there would be no enhanced predictive power from knowing that the random person also answered option 2 to Qb.

My question is, what would be the best population to compare to?

  1. I could compare to the proportion of people who answered Qa option 1 of all those who answered Qa (so this would include people not recorded in the above comparison matrix - those who answered Qa but not Qb)
  2. Or I could compare just to those who answered both Qa and Qb (ie. the sum of the above matrix)

If the questions people answered were a random subset of all questions then there should be no difference, but there is some selection bias as people could choose to skip questions they didn't want to answer.

  • I wonder if you would need to take into account the test-writing process with your probabilities. Four question multiple-choice tests are typically written with an "obvious" wrong, a less obvious wrong, then two good choices where one is more correct than the other in the context of the question. As a result, the expected probabilities for each response is not .25, .25, .25, .25 but something more like .05, .15, .25,.45 – Tavrock Mar 16 '17 at 5:21

Your Answer

 
discard

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Browse other questions tagged or ask your own question.