2
$\begingroup$

I have a very large number of true/false questions. I want to take a sample of these questions and use them to test a subject. I would like to infer the proportion of questions out of the larger set that the subject will answer correctly, based on the number the subject answered correctly in the sample.

Would it be a valid approach to model the subject's answers on a sample of questions from the full set as a Binomial experiment, assuming that the sample questions are selected randomly? If so, what assumptions am I making in using this model?

It seems to me that I would be required to assume that the subject's answers to each question is in some way fixed, so that, for a particular question, she would choose the same answer regardless of when she was asked.

Improve this question
$\endgroup$
2
  • $\begingroup$ I tried to answer, but should have read the question more carefully. Sorry $\endgroup$
    – Russ Lenth
    Commented Jul 31, 2014 at 15:41
  • $\begingroup$ I think I managed to repair my answer now so that it fits your situation. $\endgroup$
    – Russ Lenth
    Commented Jul 31, 2014 at 16:23

1 Answer 1

Reset to default
2
$\begingroup$

The binomial model is not correct, unless all the test questions have the same probability of being answered correctly. To see why, consider an extreme case where there are 100 questions in the pool. Suppose there is one question that she will definitely get right, and the other 99 she will definitely get wrong (I guess these are perfect trick questions). If the subject is given 10 questions at random, her score is $0$ with probability $\frac9{10}$ (when the one nice question is excluded) and $0.1$ with probability $\frac{1}{10}$ (when the one nice question is included) -- a multiple of a single Bernoulli trial. So her observed proportion, $X$, is a draw from a distribution with mean $0.1\times\frac{1}{10}=0.01$ and variance $0.01\times\frac9{10}=0.009$. But under the binomial model where each question has success probability $0.01$, then $X$ has the same mean $0.01$ but variance $0.01\times0.99/10=0.00099$. This is about one tenth the variance as the other situation.

If you have just one subject, I don't see that it is possible to estimate the variance, though with scenarios like the above it may be possible to find an upper bound, thus an extremely conservative approach. However, if you have several subjects, then you'll get scores $X_1,X_2,\ldots,X_n$ and you can use a simple $t$ procedure.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.