5
$\begingroup$

I have run a survey with many respondents. There was a pool of questions, and each respondent was asked 3 of the questions (chosen uniformly at random and independently for each respondent).

Now I'm interested in comparing the fraction of respondents who answered question $i$ correctly to the fraction who answered question $j$ correctly. What is the best methodology to do this?

Keep in mind that a few of the respondents will have been asked both questions $i$ and $j$ (for them, this was a within-subjects design), while many will have been asked only one or the other (for this subclass of respondents, it is more akin to a between-subjects design).

Option #1. I could count the total number of respondents who answered question $i$ and the number who answered it correctly to get the fraction who answered question $i$ correctly. Similarly, I could compute the fraction of respondents who answered question $j$ correctly. I could then compare these two numbers. Basically, I'd just be ignoring the fact that some users were asked both questions; don't worry, be happy. This is sort of a hybrid between a within-subjects/between-subjects design.

Option #2. I could filter out all the respondents who were asked both questions $i$ and $j$. Of the remaining respondents who were asked either question $i$ or question $j$ (but not both), I could then compute the fraction who answered question $i$ correctly and the fraction who answered question $j$ correctly. This is essentially converting this to a between-subjects design.

Option #3. I could keep only the answers from respondents who saw both questions, and then compare using a standard test for within-subjects designs. The downside is that I will have filtered out most of the responses; probably only a few respondents will have been randomly selected to answer both questions.

Option #4. I could do something else (you tell me what).

Do you have any advice? Or there are any pitfalls with any of these? Should any of these options be ruled out? Perhaps there is a way to combine the best of both worlds, and combine both Option #2 and Option #3 in some clever way?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
3
$\begingroup$

Unless you have a reason to believe that being exposed to question i would affect their ability to answer question j correctly, or vice versa, I think Option 1 is the best approach. I wouldn't view this as a "hybrid" approach, you're simply reporting how likely people were to answer each question correctly, which is the question you're trying to answer.

If you were a calculus professor who gave out three different versions of the final exam each quarter, each pulling from a large pool of questions, and wanted to know which questions were the hardest and which were the easiest, you would probably just look at the right/wrong answers by question and not worry about whether a student saw certain questions together or not. Not unless you had endless time and resources, or unless you had a specific hypothesis about how they interact.

From a statistical standpoint, I know there are different statistical tests for between- and within-subjects designs. While I am not well-versed in these, my layman's knowledge is that a within-subjects design would allow you more statistical power because you reduce the variance associated with individual differences between people. In your situation, since a much smaller number are exposed to both simultaneously, this doesn't actually give you an advantage. I would simply ignore the within-subjects element and treat it as between-subjects if you need to do any statistical tests.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Thanks, Jonathan. Sounds like a pragmatic answer. One aspect that I wonder about: it does not take into account all of the information (since it does not take advantage of the variance-reduction for the students who saw the same questions). Somehow it feels slightly wasteful to throw away this additional information. Is there a reasonable way to take this additional information into account? $\endgroup$
    – D.W.
    Commented Nov 18, 2011 at 5:31
  • $\begingroup$ What really matters is whether the evidence is sufficient for your practical purpose... it's always difficult to leave additional information on the cutting room floor, but sometimes that additional information won't help you make a better decision. I can't think of how that info would contribute to a different way of stack ranking the difficulty of exam questions, given that in your scenario everybody sees a small subset. $\endgroup$
    – Jonathan
    Commented Nov 19, 2011 at 7:38
  • 1
    $\begingroup$ The only advantage I can think of would be if you wanted to go further in identifying the likelihood that i is a harder question than j. This is territory where I'm really shaky, but there might be some way of constructing a Bayesian framework where you combine two different types of evidence (matched and unmatched comparisons). But I can't see that being very valuable... it still won't change the rankings, only your level of certainty of your hypothesis tests. And given that only a small number of people have seen both, I would imagine that marginal certainty to be of negligible value. $\endgroup$
    – Jonathan
    Commented Nov 19, 2011 at 7:42
  • $\begingroup$ Hi @DW, was there anything else you were looking for in your question? $\endgroup$
    – Jonathan
    Commented Dec 6, 2011 at 19:47
  • $\begingroup$ Hi @Jonathan, thanks for the helpful and practical answer! (+1, accepted) The main addition I would be interested in would be a way to squeeze out all of the available information out of the data: I don't have as much data as I wish I had, so every last bit helps. If it increases the level of confidence/certainty in my results (e.g., in the hypothesis tests), that would be sufficiently worthwhile to be worth doing, if there was a way to do it. $\endgroup$
    – D.W.
    Commented Dec 7, 2011 at 3:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.