I came across a question that I have no clue how to answer:

Suppose you are designing a diagnostic multiple choice quiz with the goal of distinguishing learners who have mastered a concept (P(correct|mastery) >= 1/2) from learners who are randomly guessing (P(correct|~mastery) = 1/4).

What is the minimum number of questions necessary to be able to declare that a learner has mastered a concept with at least 95% probability?

Assume statistical independence for the different questions, and P(mastery) = 1/2.


In Bayesian terms we could have two models ($M_1$ for having mastery, and $M_2$ for not having mastery). Using your assumption, the model priors are $p(M_1) = p(M_2) = 1/2$.

Initially we have a odds ratio of 1:1, ie, $\frac{p(M_1)}{p(M_2)} = 1$.

What you like to know is how many $c$ correct answers in $N$ questions do we need to answer to have the ratio of the model posteriors $\frac{p(M_1|c)}{p(M_2|c)} \geq 19$, since 95% means a 19:1 odds.

By Bayes theorem: $p(M_i|c) = \frac{p(c|M_i)p(M_i)}{p(c)}$.

The likelihoods $p(c|M_i)$ are binomials where you plug your initial answering probabilities accordingly for $M_1$ and $M_2$. By dividing both posteriors, $p(c)$ and $p(M_i)$ simplify and you get your formula.

  • $\begingroup$ Thanks, very clear. So the result is 5 based on (1/2)^c/(1/4)^c >= 19. I guess there is an implication that a learner needs to answer all question correctly in order to figure out minimum number of question. $\endgroup$ – Shengjie Zhang Sep 7 '17 at 5:25
  • $\begingroup$ Not quite. Notice that $N$ is necessary to consider if $c$ is low or high. The equation would simplify to $(1/2)^N / [(1/4)^c * (3/4)^{N-c} ] \geq 19$ $\endgroup$ – jpneto Sep 7 '17 at 9:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.