2
$\begingroup$

I came across a question that I have no clue how to answer:

Suppose you are designing a diagnostic multiple choice quiz with the goal of distinguishing learners who have mastered a concept $(P(\text{correct}|\text{mastery}) \ge 1/2)$ from learners who are randomly guessing $(P(\text{correct}|\text{~mastery}) = 1/4)$.

What is the minimum number of questions necessary to be able to declare that a learner has mastered a concept with at least 95% probability?

Assume statistical independence for the different questions, and P(mastery) = 1/2.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

In Bayesian terms we could have two models ($M_1$ for having mastery, and $M_2$ for not having mastery). Using your assumption, the model priors are $p(M_1) = p(M_2) = 1/2$.

Initially we have a odds ratio of 1:1, ie, $\frac{p(M_1)}{p(M_2)} = 1$.

What you like to know is how many $c$ correct answers in $N$ questions do we need to answer to have the ratio of the model posteriors $\frac{p(M_1|c)}{p(M_2|c)} \geq 19$, since 95% means a 19:1 odds.

By Bayes theorem: $p(M_i|c) = \frac{p(c|M_i)p(M_i)}{p(c)}$.

The likelihoods $p(c|M_i)$ are binomials where you plug your initial answering probabilities accordingly for $M_1$ and $M_2$. By dividing both posteriors, $p(c)$ and $p(M_i)$ simplify and you get your formula.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks, very clear. So the result is 5 based on (1/2)^c/(1/4)^c >= 19. I guess there is an implication that a learner needs to answer all question correctly in order to figure out minimum number of question. $\endgroup$
    – Shengjie Zhang
    Commented Sep 7, 2017 at 5:25
  • $\begingroup$ Not quite. Notice that $N$ is necessary to consider if $c$ is low or high. The equation would simplify to $(1/2)^N / [(1/4)^c * (3/4)^{N-c} ] \geq 19$ $\endgroup$
    – jpneto
    Commented Sep 7, 2017 at 9:46

Not the answer you're looking for? Browse other questions tagged or ask your own question.