I have run a survey with many respondents. There was a pool of questions, and each respondent was asked 3 of the questions (chosen uniformly at random and independently for each respondent).
Now I'm interested in comparing the fraction of respondents who answered question $i$ correctly to the fraction who answered question $j$ correctly. What is the best methodology to do this?
Keep in mind that a few of the respondents will have been asked both questions $i$ and $j$ (for them, this was a within-subjects design), while many will have been asked only one or the other (for this subclass of respondents, it is more akin to a between-subjects design).
Option #1. I could count the total number of respondents who answered question $i$ and the number who answered it correctly to get the fraction who answered question $i$ correctly. Similarly, I could compute the fraction of respondents who answered question $j$ correctly. I could then compare these two numbers. Basically, I'd just be ignoring the fact that some users were asked both questions; don't worry, be happy. This is sort of a hybrid between a within-subjects/between-subjects design.
Option #2. I could filter out all the respondents who were asked both questions $i$ and $j$. Of the remaining respondents who were asked either question $i$ or question $j$ (but not both), I could then compute the fraction who answered question $i$ correctly and the fraction who answered question $j$ correctly. This is essentially converting this to a between-subjects design.
Option #3. I could keep only the answers from respondents who saw both questions, and then compare using a standard test for within-subjects designs. The downside is that I will have filtered out most of the responses; probably only a few respondents will have been randomly selected to answer both questions.
Option #4. I could do something else (you tell me what).
Do you have any advice? Or there are any pitfalls with any of these? Should any of these options be ruled out? Perhaps there is a way to combine the best of both worlds, and combine both Option #2 and Option #3 in some clever way?