# How to determine sample size

I came across a question that I have no clue how to answer:

Suppose you are designing a diagnostic multiple choice quiz with the goal of distinguishing learners who have mastered a concept (P(correct|mastery) >= 1/2) from learners who are randomly guessing (P(correct|~mastery) = 1/4).

What is the minimum number of questions necessary to be able to declare that a learner has mastered a concept with at least 95% probability?

Assume statistical independence for the different questions, and P(mastery) = 1/2.

In Bayesian terms we could have two models ($M_1$ for having mastery, and $M_2$ for not having mastery). Using your assumption, the model priors are $p(M_1) = p(M_2) = 1/2$.
Initially we have a odds ratio of 1:1, ie, $\frac{p(M_1)}{p(M_2)} = 1$.
What you like to know is how many $c$ correct answers in $N$ questions do we need to answer to have the ratio of the model posteriors $\frac{p(M_1|c)}{p(M_2|c)} \geq 19$, since 95% means a 19:1 odds.
By Bayes theorem: $p(M_i|c) = \frac{p(c|M_i)p(M_i)}{p(c)}$.
The likelihoods $p(c|M_i)$ are binomials where you plug your initial answering probabilities accordingly for $M_1$ and $M_2$. By dividing both posteriors, $p(c)$ and $p(M_i)$ simplify and you get your formula.
• Not quite. Notice that $N$ is necessary to consider if $c$ is low or high. The equation would simplify to $(1/2)^N / [(1/4)^c * (3/4)^{N-c} ] \geq 19$ – jpneto Sep 7 '17 at 9:46