Data analysis for mixed within-subjects/between-subjects design? I have run a survey with many respondents.  There was a pool of questions, and each respondent was asked 3 of the questions (chosen uniformly at random and independently for each respondent).
Now I'm interested in comparing the fraction of respondents who answered question $i$ correctly to the fraction who answered question $j$ correctly.  What is the best methodology to do this?
Keep in mind that a few of the respondents will have been asked both questions $i$ and $j$ (for them, this was a within-subjects design), while many will have been asked only one or the other (for this subclass of respondents, it is more akin to a between-subjects design).
Option #1. I could count the total number of respondents who answered question $i$ and the number who answered it correctly to get the fraction who answered question $i$ correctly.  Similarly, I could compute the fraction of respondents who answered question $j$ correctly.  I could then compare these two numbers.  Basically, I'd just be ignoring the fact that some users were asked both questions; don't worry, be happy.  This is sort of a hybrid between a within-subjects/between-subjects design.
Option #2. I could filter out all the respondents who were asked both questions $i$ and $j$.  Of the remaining respondents who were asked either question $i$ or question $j$ (but not both), I could then compute the fraction who answered question $i$ correctly and the fraction who answered question $j$ correctly.  This is essentially converting this to a between-subjects design.
Option #3. I could keep only the answers from respondents who saw both questions, and then compare using a standard test for within-subjects designs.  The downside is that I will have filtered out most of the responses; probably only a few respondents will have been randomly selected to answer both questions.
Option #4. I could do something else (you tell me what).
Do you have any advice?  Or there are any pitfalls with any of these?  Should any of these options be ruled out?  Perhaps there is a way to combine the best of both worlds, and combine both Option #2 and Option #3 in some clever way?
 A: Unless you have a reason to believe that being exposed to question i would affect their ability to answer question j correctly, or vice versa, I think Option 1 is the best approach. I wouldn't view this as a "hybrid" approach, you're simply reporting how likely people were to answer each question correctly, which is the question you're trying to answer. 
If you were a calculus professor who gave out three different versions of the final exam each quarter, each pulling from a large pool of questions, and wanted to know which questions were the hardest and which were the easiest, you would probably just look at the right/wrong answers by question and not worry about whether a student saw certain questions together or not. Not unless you had endless time and resources, or unless you had a specific hypothesis about how they interact.
From a statistical standpoint, I know there are different statistical tests for between- and within-subjects designs. While I am not well-versed in these, my layman's knowledge is that a within-subjects design would allow you more statistical power because you reduce the variance associated with individual differences between people. In your situation, since a much smaller number are exposed to both simultaneously, this doesn't actually give you an advantage. I would simply ignore the within-subjects element and treat it as between-subjects if you need to do any statistical tests.
