Determining sample size from a Bayesian problem

Question

I was following up on this problem when it seemed to me that the correct answer was ambiguous. I will not re-state the problem since it can be found in the original link. I will use the same notation conventions as the accepted answer, namely that $M_1$ will designate the model for having mastery and$M_2$ for not having mastery. $N$ and $c$ indicate the number of questions and correct answers, respectively.

To be more explicit, where I am having difficulty, I'll pickup from where the accepted answer finished. We have the ratio of model posteriors:

$\frac{p(M_1|c)}{p(M_2|c)} \geq 19 $

By plugging in using Bayes theorem, we have:

$\frac{p(c | M_1)p(M_1)}{p(c | M_2)p(M_2)} \geq 19 $

The likelihoods $p(M_i|c)$ are binomials and $p(M_i)$ are the given priors. By substituting with numerical values given in the problem, we get:

$ \frac{{N \choose c} (1/2)^c(1 - 1/2)^{N-c}\times (1/2)} {{N \choose c} (1/4)^c (1-1/4)^{N-c} \times (1/2)} \geq 19 $

This can be simplified to:

$ \frac{(1/2)^N}{(1/4)^c \times (3/4)^{N-c}} \geq 19 $.

However, at this point, we cannot directly solve for $N$, since $c$ is also a variable. What is the correct approach for determining the smallest possible value of $N$?

Answer 1

Best-case Scenario

The smallest possible value of $N$ where we could be $95\%$ confident that the student has "mastered" a concept (meaning that their probability of answering a question correctly is $1/2$) would correspond to picking $c = N$, because $P(M_1 | c)$ is an increasing function of $c$, and $c$ is constrained by $c \le N$. In this case, $$ \frac{(1/2)^N}{(1/4)^c \times (3/4)^{N-c}} = \frac{(1/2)^N}{(1/4)^N} = 2^N $$ and the smallest value of $N$ that makes $2^N \ge 19$ is $N = 5$.

Anti-best-case Scenario

If we wanted to be $95\%$ confident that a student has not mastered the material, then the best-case scenario would be $c = 0$ where the student misses every question. Then we have $$ \frac{(1/2)^N}{(1/4)^c \times (3/4)^{N-c}} = \frac{(1/2)^N}{(3/4)^N} = (2/3)^N $$ and the smallest value of $N$ for which $(2/3)^N \le 1/19$ is $N = 8$, in which case we can be $95\%$ confident that the student has not mastered the material.

Worst-case Scenario

Assuming that the student has a fixed probability $\delta$ of getting the questions right, in the limit as $N \to \infty$ we have $c \approx \delta N$. Taking $\log_2$ of both sides of the desired inequality we have \begin{align} \frac{(1/2)^N}{(1/4)^c \times (3/4)^{N-c}} & \ge 19 \\ N \log_2(1/2) - c\log_2(1/4) - (N - c)\log_2(3/4)& \ge \log_2 (19) \\ N \log_2(1/2) + c\log_2(4) + (N - c)\log_2(4/3)& \ge \log_2 (19) \\ -N + 2c + (N - c)(2 - \log_2(3)) & \ge \log_2 (19) \\ -N + 2\delta N + (1 - \delta)(2 - \log_2(3))N & \ge \log_2 (19) \\ N[-1 + 2\delta + (1 - \delta)(2 - \log_2(3))] & \ge \log_2 (19) \\ N[1 - \log_2(3) + \log_2(3) \delta] & \ge \log_2 (19). \end{align} Because the expression $1 - \log_2(3) + \log_2(3) \delta$ is 0 when $$\delta = \frac{\log_2 (3) - 1}{\log_2 (3)} \approx 0.36907,$$ we can never be $95\%$ confident either way even if $N \to \infty$ for a student that gets exactly this fraction of questions correct. This means that there is no universally valid value $N$ that will always ensure $95\%$ confidence either that the student is a master or is not a master. Notice that this value is very close to the midpoint between $1/2$ and $1/4$, which is $0.375$. Intuitively, it is difficult to determine whether a student that gets $\approx 37\%$ of questions correct is a master with probability $P(\text{correct}) = 1/2$ or not a master with probability $P(\text{correct}) = 1/4$.